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PREFACE 
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toward  the  evaluation  of  crew  workload  assessment  techniques  for  aircraft  certification.  This 
study  was  conducted  as  a  joint  effort  by  the  two  major  U.S.  manufacturers  of  commercial 
transport  airplanes:  Douglas  Aircraft  Company  and  Boeing  Commercial  Airplanes.  The 
primary  purpose  of  this  volume  is  to  report  the  results  of  the  contract  effort.  The  objective  of 
this  contract  was  to  provide  assessment  criteria  to  enable  the  FAA  to  evaluate  workload 
measurement  plans  for  crew  size  substantiation  and  workload  acceptability  during  aircraft 
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1.0  INTRODUCTION 

In  1981  the  Presidents  task  force  on  crew  complement  reported  on  workload 
assessment  techniques  employed  in  aircraft  certification  programs.  In  general,  the 
workload  measures  and  test  domain  employed  in  recent  certification  programs  were 
judged  to  be  "state  of  the  art";  however,  a  number  of  improvements  have  been 
recommended  for  future  efforts  (McLucas,  Drinkwater,  and  Leaf,  1981).  The 
recommendations  include: 

(a)  Improving  subjective  measurement  methods, 

(b)  Studying  crew  performance  under  a  variety  of  conditions;  line  operations 
(full-mission)  simulation  using  selected  line  pilots  used  in  conducting  these 
studies, 

(c)  Consulting  with  qualified  line  pilots  in  the  area  of  workload  evaluation, 

(d)  Evaluating  the  impact  on  crew  workload  of  selected  minimum  equipment 
list  (MEL)  items. 

The  "state-of-the-art"  technique  referred  to  by  President's  task  force  is  Timeline  Analysis. 
Timeline  analysis,  based  on  micro-motion  techniques  and  borrowed  from  Industrial 
Engineering,  computes  workload  as  a  ratio  of  time  required  to  complete  necessary  tasks 
as  a  fraction  of  time  available.  Timeline  analysis  is  an  analytic  technique  which  does  not 
require  "pilot  in  the  loop"  measurement.  Workload  measurement  techniques  can  be 
classified  into  the  following  categories: 

(a)  Subjective  ratings, 

(b)  Physiological  recordings, 

(c)  Performance  in  the  piloting  task. 

More  recently  subjective  judgements  have  been  used  for  evaluating  the  acceptability  of 
flight  deck  workload  during  certification.  Recent  improvements  in  workload 
measurement  science,  however,  have  led  to  the  perception  that  other  methods  for 
evaluating  workload  exist,  and  perhaps  they  are  ready  to  be  employed  in  aircraft 
certification  applications.  The  purpose  of  this  contract  is  to  identify  valid  and  reliable 
workload  measures,  and  provide  a  methodology  for  the  correct  application  of  those 
measures  in  a  high  fidelity  aircraft  flightdeck  environment. 

A  list  of  acceptable  workload  assessment  techniques  and  scenarios  for  aircraft 
certification  can  be  derived  upon  application  of  a  selection  criteria  and  a  set  of  boundary 
conditions  for  the  criteria.  The  minimum  criteria  for  acceptability  of  a  workload  measure 
is  that  the  measure  is  valid,  reliable,  and  applicable  to  aircraft  certification.  The  boundary 
condition  should  consist  of  a  set  of  statements  which  identify  the  proper  domain  in  which 
the  measure  is  valid  (measures  what  it  is  believed  to  measure)  and  reliable  (stable  or 
consistent  in  what  it  measures).  Applicable  assessment  techniques  are  those  which  are 
safe,  cost-effective,  can  be  applied  in  a  timely  fashion,  and  can  be  implemented  in  the 
flight  deck  of  a  commercial  aircraft.  The  guidelines  which  are  presented  in  Volume  Two 
are  an  attempt  to  specify  the  important  boundary  conditions  for  workload  assessment  in 
an  aircraft  certification  program. 

1.1  OBJECTIVES 

The  primary  purpose  of  these  guidelines  is  to  enable  the  FAA  and  USAF  to  evaluate 
workload  assessment  plans.  This  is  to  be  accomplished  by  providing  a  recommended: 
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(a)  Process  for  evaluating  candidate  workload  measurement  techniques  and 
task  scenarios, 

(b)  Usage  of  the  workload  measurement  concepts, 

(c)  List  of  measures  which  have  exhibited  evidence  of  validity  and  reliability  in 
the  assessment  of  civil  transport  workload. 

Several  specific  objectives  are  identified  to  facilitate  the  evaluation  of  a  workload 
certification  plan  and  are  as  follows: 

(a)  Provide  guidelines  for  evaluating  a  proposed  aircraft  workload  certification 
plan  that  will  enable  the  FAA  to  insure  that  the  workload  criteria  specified  in 
FAR  15.2523,  Appendix  D  are  adequately  considered, 

(b)  Provide  guidelines  on  how  to  apply  workload  assessment  techniques  in 
aircraft  certification, 

(c)  Provide  guidelines  for  evaluating  the  adequacy  of  the  proposed  workload 
measures  and  scenarios  included  in  a  workload  certification  plan, 

(d)  Provide  examples  of  evaluation  criteria  for  the  determination  of  acceptable 
workload  measures  and  scenarios, 

(e)  Provide  a  data  base  to  aid  the  FAA  and  USAF  in  location  of  factual 
information  about  workload  measures  suitable  for  aircraft  certification. 

1.2  BACKGROUND 

Requirements  to  assess  transport  aircraft  crew  workload  have  developed  as  a  means  of 
assuring  that  the  task  demands  imposed  on  the  aircrew  will  not  exceed  the  crew's  ability 
to  respond  to  them  in  a  safe  and  timely  fashion.  The  requirements  for  assessment  of 
crew  workload  are  specified  in  FAR  25.1523  Appendix  D  and  in  FAR  25.771  (see  Table 
1.2-1). 

A  distinction  should  be  made  between  the  workload  experienced  by  the  crew  and  the 
task  demands  imposed  on  the  crew.  This  distinction  is  often  characterized  as  a 
separation  between  the  demands  placed  on  the  crew  member  by  the  machine  and  the 
crew  member's  response  to  those  demands.  The  demands  of  the  task  are  dependent 
on  the  requirements  of  the  system,  the  effort  (workload)  put  forth  by  the  crew  member  is 
dependent  on  his  perception  of  the  task  demands,  and  the  results  of  that  combined 
man/machine  effort  determines  the  system  performance. 

The  distinction  between  perceived  mental  workload  and  the  perceived  workload  in 
general,  has  not  always  been  easy  to  make.  Although  early  investigators  such  as 
Cooper  and  Harper  (1969)  have  distinguished  between  physical  and  mental  effort, 
completely  satisfying  methods  for  separating  them  have  not  been  developed.  Physical 
workload  is  computed  in  terms  of  the  actual  movements  (i.e.,  eye  and  hand  movements) 
needed  to  execute  a  procedure.  Mental  workload  is  assumed  to  occur  when  a  human 
operator  performs  higher  order  functions  such  as  perception,  information  processing,  or 
decision-making.  Excessive  mental  workload  will  manifest  itself  in  the  system  by  longer 
operator  processing  time,  shedding  of  tasks,  increased  errors,  performance 
decrements,  and  motivational  lapses. 


TABLE  1.2-1.  FEDERAL  AVIATION  REGULATION  REQUIREMENTS 


REQUIREMENT 


FAR  25.771  PILOT  COMPARTMENT 

(a)  Each  pilot  compartment  and  its  equipment  must  allow  the  minimum  flight  crew 
(established  under  25.1523)  to  perform  their  duties  without  unreasonable 
concentration  or  fatigue. 


FAR  25.1523  MINIMUM  FLIGHTCREW 

The  minimum  flight  crew  must  be  established  so  that  it  is  sufficient  for  safe  operation, 
considering  -  - 

(a)  The  workload  on  individual  crew  members; 

(b)  The  accessibility  and  ease  of  operation  of  necessary  controls  by  the  appropriate 
crewmember;  and 

(c)  The  kind  of  operation  authorized  under  25. 1 525. 

The  criteria  used  in  making  the  determinations  required  by  this  section  are  set  forth  in 
Appendix  D. 


FAR  25  APPENDIX  D 

Criteria  for  determining  minimum  flight  crew.  The  following  are  considered  by  the 
Agency  in  determining  the  minimum  flight  crew  under  25.1523: 

a.  Basic  workload  function.  The  following  basic  workload  functions  are  considered: 

(1)  Flight  path  control 

(2)  Collision  avoidance 

(3)  Navigation 

(4)  Communications 

(5)  Operation  and  monitoring  of  aircraft  engines  and  systems 

(6)  Command  decisions 

b.  Workload  factors.  The  following  workload  factors  are  considered  significant  when 
analyzing  and  demonstrating  workload  for  minimum  flight  crew  determination: 
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TABLE  1.2-1.  FEDERAL  AVIATION  REGULATION  REQUIREMENTS 

(Continued) 

(1)  The  accessibility,  ease,  and  simplicity  of  operation  of  all  necessary  flight, 
power  and  equipment  controls,  including  emergency  fuel  shutoff  valves, 
electrical  controls,  electronic  controls,  pressurization  system  controls,  and 
engine  controls. 

(2)  The  accessibility  and  conspicuity  of  all  necessary  instruments  and  failure 
warning  devices  such  as  fire  warning,  electrical  system  malfunction,  and 
other  failure  or  caution  indicators.  The  extent  to  which  such  instruments  or 
devices  direct  the  proper  corrective  action  is  also  considered. 

(3)  The  number,  urgency,  and  complexity  of  operating  procedures  with 
particular  consideration  given  to  the  specific  fuel  management  schedule 
imposed  by  center  of  gravity,  structural  and  other  considerations  of  an 
airworthiness  nature,  and  to  the  ability  of  each  engine  to  operate  at  all  time 
from  single  tank  or  source  which  is  automatically  replenished  if  fuel  is  also 
stored  in  other  tanks. 

(4)  The  degree  and  duration  of  concentrated  mental  and  physical  effort 
involved  in  normal  operation  and  in  diagnosing  and  coping  with 
malfunctions  and  emergencies. 

(5)  The  extent  of  required  monitoring  of  the  fuel,  hydraulic,  pressurization, 
electrical,  electronic,  deicing,  and  other  systems  while  enroute. 

(6)  The  actions  requiring  a  crewmember  to  be  unavailable  at  his  assigned  duty 
station,  including:  observation  of  systems,  emergency  operation  of  any 
control,  and  emergencies  in  any  compartment. 

(7)  The  degree  of  automation  provided  in  the  aircraft  systems  to  afford  (after 
failure  or  malfunctions)  automatic  crossover  or  isolation  of  difficulties  to 
minimize  the  need  for  flight  crew  action  to  guard  against  loss  of  hydraulic 
or  electric  power  to  flight  controls  or  to  other  essential  systems. 

(8)  The  communications  and  navigation  workload. 

(9)  The  possibility  of  increased  workload  associated  with  any  emergency  that 
may  lead  to  other  emergencies. 

(10)  Incapacitation  of  a  flight  crewmember  whenever  the  applicable  operating 
rule  requires  a  minimum  flight  crew  of  at  least  two  pilots. 

c.  Kind  of  operation  authorized.  The  determination  of  the  kind  of  operation 
authorized  requires  consideration  of  the  operating  rules  under  which  the  airplane 
will  be  operated.  Unless  an  applicant  desires  approval  for  a  more  limited  kind  of 
operation,  it  is  assumed  that  each  airplane  certificated  under  this  Part  will  operate 
under  IFR  conditions. 
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1.3  SCOPE 


Valid,  reliable,  and  applicable  techniques  for  workload  assessment  are  addressed  in  the 
following  contract.  Issues  of  overload  and  underload  are  addressed  only  to  the  degree 
that  those  states  can  be  inferred  using  valid  and  reliable  workload  measurement 
techniques,  but  no  attempt  is  made  to  create  those  situations  during  simulation  so  that 
the  candidate  measures  can  be  evaluated  for  their  ability  to  identify  those  states.  Fatigue 
is  another  issue  related  to  workload  that  is  not  specifically  addressed  in  the  simulation 
studies  conducted  to  evaluate  the  candidate  workload  measures. 

1.3.1  PROGRAM  CONSTRAINTS 

Owing  to  the  large  number  of  possible  workload  measures  which  could  be  reviewed  and 
evaluated,  certain  self-imposed  limitations  were  outlined  to  insure  adequate  resources 
would  be  available  for  a  reasonable  quality  evaluation  of  the  candidate  measures.  The 
following  limitations  were  outlined: 

(a)  Workload  measures  were  selected  from  those  which  already  existed.  A 
candidate  measure  had  to  have  published  evidence  of  validity  and 
reliability. 

(b)  The  "best"  measures  were  evaluated  from  each  of  three  categories: 
Subjective,  Physiological,  and  Performance  workload  assessment 
techniques.  "Best"  was  defined  as  the  most  valid,  reliable,  and  applicable 
to  aircraft  certification.  The  number  of  measures  to  be  evaluated  was 
limited  by  the  available  resources. 

(c)  Only  measures  which  were  suitable  for  high  fidelity  simulation  or  flight  test 
were  evaluated. 

(d)  Measures  were  evaluated  in  a  civil  transport  environment  (as  opposed  to 
military  environments).  The  types  of  task  demands  addressed  in  scenario 
selection  were  identified  by,  but  not  limited  to,  the  functions  and  factors  in 
FAR  25.1523  Appendix  D.  (The  results  obtained  from  scenarios  which  are 
common  with  military  task  demands  will  be  valid  for  military  applications). 

(e)  The  issues  of  underload  and  fatigue  were  not  examined. 

(f)  A  list  of  "acceptable"  workload  measures  was  developed,  where 
"acceptable"  was  defined  as  a  measure  which  had  evidence  of  both  validity 
and  reliability  with  representative  civil  transport  workload.  An  acceptable 
measure  was  presumed  to  be  valid  and  re  iable  for  only  specific  types  of 
workload.  The  designation  of  acceptabe  did  not  imply  it  could  be 
employed  in  every  condition. 

Owing  to  limitations  of  funding,  only  the  pilot  was  used  as  a  subject  in  the  current 
investigations.  Even  though  the  simulation  test  bed  was  a  three  crew  aircraft  (Captain, 
First  Officer,  and  Flight  Engineer)  the  workload  assessment  techniques  are  being 
considered  for  use  in  certification  of  newer,  two-person,  aircraft.  Newer  generations  of 
aircraft  utilizing  two  crew  members  usually  divide  flightdeck  responsibilities  into  pilot 
flying  (PF)  and  pilot  not  flying  (PNF).  PF  responsibilities  are  primarily  aircraft  control  and 
navigation.  The  PNF  is  responsible  for  communication,  system  monitoring,  and  any 
tasks  delegated  by  the  PF.  The  current  investigation  manipulated  workload  so  that  both 
PF  and  PNF  task  demands  were  represented  by  the  pilot. 

1.3.2  DESIGN  VERSUS  CERTIFICATION 

Workload  measurement  is  most  effective  when  the  new  aircraft  design  is  complete,  a 
high  fidelity  simulation  is  available,  airplane  development  testing  is  completed,  and  the 
crew  is  fully  trained.  A  high  fidelity  simulator  or  flight  test  aircraft  can  present 
representative  task  demands  to  a  crew,  thereby  maximizing  the  probability  that 
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actual  workload  levels  experienced  during  line  service.  Besides  its  role  as  a  check  on 
the  final  design  to  assure  that  workload  is  acceptable,  workload  assessment  can  also  be 
employed  in  the  design  of  an  aircraft  or  of  one  of  its  subsystems.  Caution  should  be 
exercised  when  applying  workload  assessment  techniques  during  design  due  to  the  lack 
of  system  integration  in  the  to-be-evaluated  flight  deck.  The  combination  of  systems  can 
produce  increases  or  decreases  in  workload  that  were  absent  during  individual  system 
evaluation.  Analytic  techniques  do  enable  the  designer  to  make  relatively  conservative 
estimates  of  what  the  crew’s  task-demands  will  be  so  that  it  will  be  virtually  certain  that 
the  actual  workload  experienced  by  the  crew  will  be  acceptable. 

There  are  workload  assessment  techniques  that  might  be  appropriate  for  use  in  the 
design  phase  that  were  not  considered  in  this  effort  because  they  are  not  yet  applicable 
in  the  certification  flight  environment.  It  is  the  manufacturers'  responsibility  to  select  and 
propose  appropriate  workload  assessment  techniques  for  certification. 

The  nature  of  the  equipment  change  should  determine  the  workload  assessment 
method  selected  for  certification.  Each  specific  type  design  may  generate  different  types 
of  workload,  and  therefore,  different  workload  assessment  techniques  may  be  required. 
Not  all  measures  are  equally  sensitive  to  the  different  types  of  workload  (e.g.,  physical 
vs.  mental). 

1.3.3  CERTIFICATION  APPLICATIONS 

The  most  relevant  consideration  when  evaluating  a  workload  assessment  plan  is  that  the 
proposed  workload  techniques  are  adequate  for  evaluating  the  anticipated  workload  for 
the  new  flight  deck. 

In  the  early  days  of  commercial  jet  aircraft  (i.e.,  B-707  and  DC-8),  workload  was  primarily 
of  a  physical  nature.  When  the  commercial  flight  deck  of  smaller  aircraft  (less  than 
200,000  lbs.)  was  reduced  to  two  crew  members  (i.e.,  DC-9  and  B-737),  the  systems 
contained  a  higher  degree  of  integration.  The  evaluation  was  centered  on  the  physical 
nature  of  workload.  The  workload  evaluation  focused  on  the  question  of  the  two  crew 
members  (one  crew  member  during  the  case  of  incapacitation)  accomplishing  all  the 
needed  tasks  for  flying  the  aircraft.  Next  generations  of  aircraft  (the  glass  cockpits  of  the 
B-757/767  and  MD-88)  retained  the  two  crew  member  flightdeck,  but  the  move  to 
sophisticated  Flight  Management  Systems  introduced  new  levels  of  mental  workload  for 
evaluation  during  the  workload  certification  effort.  Finally,  the  latest  certification  efforts 
involve  traditionally  three  crew  aircraft  (MD-11  and  B-747/400)  being  stretched  for 
increased  passenger  and  range  capacity,  while  increasing  automation  levels  and 
eliminating  the  flight  engineer. 

In  the  past,  commercial  aircraft  manufacturers  have  used  analytic  techniques  and  non- 
structured  pilot  opinion  for  workload  assessment.  Analytic  techniques  are  of  particular 
value  to  the  aircraft  manufacturer  since  they  offer  both  the  potential  for  identifying  and 
correcting  workload  problems  early  in  the  design  phase  when  the  cost  of  change  is 
relatively  low,  and  a  tool  which  can  provide  data  for  certification.  One  disadvantage  to 
the  available  analytic  techniques  is  their  lack  of  fidelity  in  assessing  mental  effort.  With 
the  current  shift  of  flight  deck  design  placing  more  mental  demands  on  the  flight  crew, 
workload  assessment  has  taken  on  a  new  challenge.  The  addition  of  structured 
subjective  measures  to  traditional  objective  analyses  can  provide  information  which 
validates  the  analytic  and  simulation  based  estimates  of  physical  workload  and 
enhances  estimates  of  mental  workload. 

In  addition  to  the  consideration  that  appropriate  assessment  techniques  be  applied, 
consideration  must  also  be  given  to  valid  test  methods.  A  partial  listing  of  common 
methodological  errors  is  given  below: 
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In  addition  to  the  consideration  that  appropriate  assessment  techniques  be  applied, 
consideration  must  also  be  given  to  valid  test  methods.  A  partial  listing  of  common 
methodological  errors  is  given  below: 

(a)  Demand  characteristics  are  not  controlled  (e.g.,  hints  are  inadvertently 
given  to  the  subject  on  how  to  rate  the  workload  as  high  or  low), 

(b)  No  differences  should  exist  in  the  test  scenario  between  the  baseline  and 
to-be-certificated  aircraft,  otherwise  observed  workload  differences  could 
be  attributed  to  changes  in  the  test  scenario  instead  of  the  aircraft, 

(c)  Order  effects  of  testing  (e.g.,  learning  or  fatigue  effects)  are  not  controlled. 

1 .3.4  CONSIDERATION  OF  MILITARY  STANDARDS 

A  review  was  made  of  the  military  standard  entitled:  Human  Engineering  Requirements 
for  Military  Systems,  Equipment,  and  Facilities  (MIL-H-46855A)  and  the  draft  military 
standard  entitled:  Human  Engineering  Requirements  for  Measurement  of  Operator 
Workload.  The  approach  taken  in  this  contract  of  a  literature  search,  workshops,  and 
simulation  testing  provides  a  methodology  appropriate  to  addressing  the  issues  raised  in 
the  Military  Standards  documents. 

1.3.5  RELATIVE  VERSUS  ABSOLUTE  MEASUREMENT 

Workload  assessment  for  certification  relies  on  a  relative  comparison  of  workload  levels. 
Typically  workload  is  compared  between  the  to-be-certificated  aircraft  and  a  baseline 
aircraft,  which  has  an  established  record  of  safe  performance  and  acceptable  workload. 
It  is  assumed  that  the  two  aircraft  are  being  compared  under  conditions  which  are  as 
similar  as  possible  to  insure  that  any  workload  differences  which  occur  are  due  to 
differences  in  the  aircraft  design  and  not  to  other  factors.  If  the  new  model  aircraft  has 
the  same,  or  lower  workload,  then  it  is  concluded  that  the  workload  is  acceptable  in  the 
new  model.  When  performing  a  relative  comparison  with  a  new  aircraft  design,  however, 
there  may  be  instances  when  workload  levels  exceed  the  old  design.  In  cases  such  as 
this  the  increased  workload  is  not  necessarily  unacceptable,  but  it  may  become  the 
subject  of  a  more  in-depth  workload  analysis.  These  cases  need  to  be  considered  on  a 
case-by-case  basis  with  all  of  the  operational  factors  taken  into  consideration  when 
evaluating  the  impact  of  workload  increases. 

When  measuring  pilot  workload,  or  any  other  behavioral  measure,  it  is  essential  to 
consider  the  variable  nature  of  the  data.  Behavioral  data  is  best  described  in  terms  of 
distributions,  since  individuals  bring  different  skills  to  the  task  of  flying  it  is  possible  to  get 
a  distribution  of  workload  scores  from  a  group  of  pilots.  The  "state  of  the  art"  of 
workload  science  does  not  allow  for  determination  of  a  single  score  for  the  purpose  of 
workload  assessment.  Pilot  to  pilot  variability  in  assessing  workload  is  a  consideration 
which  must  be  kept  in  mind  throughout  an  aircraft  certification  effort.  No  absolute 
measure  of  workload  ("Redline"  associated  with  an  overload  condition)  is  currently 
available  for  aircraft  certification.  Considering  the  scope  and  magnitude  of  individual 
differences  it  would  be  extremely  risky  to  use  a  single  workload  measurement  technique 
to  determine  if  flight  crew  workload  is  acceptable. 

A  number  of  factors  influence  the  ability  to  generalize  or  draw  conclusions  about 
workload  levels  made  in  a  comparative  evaluation.  It  would  not  be  appropriate  to  include 
a  detailed  discussion  of  these  factors  here,  but  a  partial  listing  of  relevant  factors 
includes: 

!a)  Representativeness  of  subject  selection, 

b)  Number  of  subjects  tested, 

c)  Fidelity  of  task  demands  or  scenarios, 
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1.4  CURRENTLY  USED  TECHNIQUES 


Today's  list  of  acceptable  workload  measures  is  likely  to  be  out  of  date  10-15  years  from 
today.  Any  list  which  is  fixed  and  cannot  be  modified  to  accommodate  the 
improvements  developed  within  the  workload  measurement  science  could  become  more 
of  an  obstacle  than  an  aid  in  certifying  the  design  of  a  new  aircraft.  For  this  reason, 
emphasis  should  be  placed  on  whether  the  most  useful  measure  was  selected  for  a 
particular  application,  rather  than  selection  of  a  measure  merely  because  it  was  familiar 
or  associated  with  a  prior  list. 

1 .5  OVERVIEW  VOLUME  ONE 

The  remainder  of  this  report  describes  the  process  undertaken  in  the  course  of  the 
contract. 

To  identify  candidate  workload  measures  for  evaluation,  a  literature  search  was 
conducted.  Selection  criteria  were  established  and  a  Fact  Matrix  developed  to  facilitate 
the  comparison  of  various  measures  with  an  empirical  record  of  validity  and  reliability. 

Two  workshops  were  conducted  to  gather  expert  opinion  regarding  the  workload 
measures  used  for  simulation  and  the  approach  to  simulation  testing. 

Finally,  the  methods  and  results  of  two  simulation  tests  conducted  at  the  NASA-Ames 
Research  Center  are  presented. 


•% 
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2.0  SELECTION  CRITERIA  FOR  WORKLOAD  ASSESSMENT  TECHNIQUES 

In  order  to  evaluate  the  utility  of  a  workload  measure  specific  criteria  regarding  the 
validity,  reliability,  and  applicability  need  to  be  established.  The  following  sections 
identify  specific  issues  that  should  be  addressed  regarding  validity,  reliability,  and 
applicability. 

2.1  VALIDITY  CRITERIA 

Validity  is  defined  as  the  capacity  of  an  assessment  technique  to  quantitatively  evaluate 
levels  of  workload.  There  are  many  types  of  validity,  each  affecting  the  ultimate 
usefulness  and  acceptability  of  a  workload  measure.  Not  to  consider  validity  when 
selecting  a  workload  measure  is  to  employ  a  measure  that  carries  greater  risk  of  giving 
spurious  results,  and  it  may  be  better  to  design  an  aircraft  with  less  information  than  with 
incorrect  information.  At  a  minimum  it  is  proposed  that  the  following  types  of  validity  be 
addressed  during  measure  selection,  testing,  and  evaluation  (Anastasi,  1968): 

(a)  CONTENT  VALIDITY  -  It  is  important  to  determine  that  the  operationally 
relevant  types  of  workload  are  being  considered.  By  focusing  on  the 
important  types  of  workload  found  in  cockpit  operations,  more  confidence 
can  be  obtained  that  the  correct  workload  assessment  techniques  will  be 
selected  and  employed. 

(b)  PREDICTIVE  VALIDITY  -  (Also  known  as  Criterion  Related  Validity)  Can 
the  measure  be  used  in  a  predictive  fashion  to  determine  levels  of 
workload?  This  is  the  most  important  type  of  validity  for  the  manufacturer 
because  it  provides  a  basis  for  making  cost/benefit  decisions  regarding 
system  design. 

(c)  CONSTRUCT  VALIDITY  -  The  construct  validity  of  a  workload  assessment 
technique  is  the  extent  to  which  the  technique  may  be  said  to  measure  the 
theoretical  construct  of  workload.  Since  workload  cannot  be  directly 
observed,  it  exists  only  as  an  theoretical  construct,  it  must  be 
demonstrated  that  the  measure  reflects  changes  in  what  would  be 
predicted  for  the  construct  of  workload.  To  have  confidence  in  a  workload 
measure,  this  connection  must  be  demonstrated  whether  workload  is 
defined  in  terms  of  task  demands  or  operator  variables.  Construct  validity 
is  not  accomplished  in  a  single  experiment  or  settled  "once  and  for  all,"  it 
required  the  gradual  accumulation  of  information  from  a  variety  of  sources. 

(d)  FACE  VALIDITY  -  Face  validity  refers  to  what  the  assessment  technique 
appears  superficiality  to  measure  and  not  necessarily  what  it  actually 
measures.  Face  validity  can  become  important  in  how  well  people  use  an 
assessment  technique.  If  pilots  or  engineers  are  asked  to  use  or 
administer  a  workload  measurement  system  that  makes  little  sense  to 
them,  their  motivation  to  follow  all  the  rules  is  likely  to  suffer. 

2.2  RELIABILITY  CRITERIA 

Reliability  is  defined  as  the  capacity  of  a  workload  measure  to  yield  similar  results  with 
repeated  usage.  How  consistently  does  the  measure  yield  the  same  answer  given  the 
same  measurement  conditions?  The  following  types  of  reliability  should  be  addressed 
when  considering  a  workload  for  test  and  evaluation  (Anastasi,  1968): 
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(a)  TEST-RETEST  RELIABILITY  -  The  most  obvious  method  for  determining 
the  reliability  of  an  assessment  method  is  to  repeat  the  test  conditions  on  a 
second  occasion,  and  evaluate  discrepancies  in  the  two  samples.  The 
reliability  can  be  quantified  by  computing  the  correlation  between  the  two 
sets  of  scores  obtained  by  the  same  persons  on  the  two  administrations  of 
the  workload  measure.  The  resulting  reliability  coefficient  can  then  be 
compared  to  established  standards  for  any  test  and  thereby  be  viewed 
with  some  objectivity. 

(b)  SPLIT-HALF  RELIABILITY  -  A  method  employed  to  determine  the 
consistency  of  an  assessment  technique,  with  regard  to  content 
sampling,  is  to  divide  the  assessment  into  comparable  halves  and 
compute  the  correlation  between  the  two  sets  of  scores. 

(c)  INTER-RATER  RELIABILITY  -  A  method  employed  to  determine  the 
consistency  of  an  assessment  technique  across  different  people,  is  to 
compute  the  similarity  of  each  pilot's  workload  score  with  every  other  pilot 
(or  some  representative  pilot  score)  in  the  same  test. 

For  the  aircraft  manufacturer  these  types  of  reliability  definitions  are  appropriate  for 
workload  measurement  when  each  is  discussed  in  terms  of  discrimination.  For  instance, 
if  an  assessment  technique  is  being  used  to  discriminate  between  high  and  low  "levels" 
of  workload,  it  should  discriminate  between  high  and  low  "levels"  the  same  way  on  a 
second  occasion. 

2.3  APPLICABILITY  CRITERIA 

Applicability  of  workload  measures  refers  to  the  extent  that  the  measure  in  question 
impacts  the  performance  of  the  transport  aircraft  mission  and  is  practical.  For  a 
workload  measure  to  be  applicable  in  a  flight  operational  environment,  it  should  satisfy  a 
number  of  requirements.  The  following  requirements  may  not  always  be  achievable  in 
an  absolute  sense,  but  they  can  be  viewed  as  guidelines: 

(a)  The  assessment  method  should  be  as  unobtrusive  as  possible.  It  should 
not  impose  an  additional  workload  on  the  crew  and  thereby  disturb  the 
very  process  that  it  is  trying  to  measure. 

(b)  The  assessment  method  should  be  as  noninterfering  as  possible.  It  should 
not  endanger  the  crew's  safety  nor  interfere  with  their  normal  duties. 

(c)  The  assessment  method  should  be  non-career  threatening  to  the  crew 
members  it  evaluates  (e.g.,  data  collected  using  physiological  measures 
should  contain  no  diagnostic  medical  information). 

(d)  The  assessment  method  should  cause  minimal  interference  with  other 
certification  flight  test  activities.  The  technique  should  be  appropriate  for 
the  specific  phase  and  objectives  of  the  certification  program.  Time 
constraints  include  the  certification  program  schedule,  production 
schedules,  and  delivery  schedules. 

(e)  Cost  associated  with  the  workload  measurement  technique  must  be 
reasonable.  Costs  may  include  equipment,  installation  and  preparation, 
time  and  schedule  impact,  flight  and  simulation,  data  reduction  and 
analyses,  and  the  documentation  of  the  results. 
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(f)  Equipment  constraints  include  such  factors  as  limited  hardware  space, 
limited  panel  space,  a  large  distance  between  pilot  and  data  collection 
hardware,  potential  signal  interference,  and  the  inability  to  change  the  flight 
deck  configuration. 


♦ 


ft 
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3.0  SUMMARIZE  KNOWLEDGE  ABOUT  WORKLOAD  MEASURES 


In  order  to  determine  the  optimal  subset  of  workload  measures  to  be  tested  in  simulation 
a  literature  review  was  conducted.  The  intent  of  the  workload  literature  review  was  to 
identify,  collect,  organize,  and  publish  a  cross  referenced  index  of  published  articles 
which  addressed  the  issue  of  workload  measurement. 

3.1  DESCRIPTION  OF  LITERATURE  REVIEW 

LITERATURE  SEARCH 

Using  the  keyword  "workload"  an  electronic  search  was  conducted  of  library  data  bases 
from  McDonnell  Douglas,  the  Boeing  Company,  NASA,  the  U.S.  Air  Force,  and 
universities  in  the  metropolitan  Los  Angeles  area.  Abstract  searches  generated  a  list  of 
document  titles  which  were  then  collected  for  evaluation.  Only  material  published  after 
1978  was  evaluated  in  the  present  effort. 

Criteria  were  generated  to  facilitate  discrimination  of  likely  documents  for  detailed  review. 
The  articles  were  sorted  according  to  the  following  criteria: 

(a)  Empirical  data  from  flight  test  or  simulation, 

(b)  Empirical  data  from  laboratory  experimentation, 

(c)  Review  article, 

(d)  Theoretical  article. 

In  addition  to  library  literature  searches,  recognized  workload  assessment  experts  were 
contacted  to  assist  in  obtaining  the  most  recent  literature.  The  results  of  the  literature 
search,  and  the  expert  summaries,  were  integrated  into  fact  matrices. 

3.2  DESCRIPTION  OF  FACT  MATRIX  DEVELOPMENT 

Matrices  were  developed  that  cross-referenced  workload  measure  by  different  types  of 
validity  and  reliability.  The  first  criterion  for  consideration  was  whether  the  technical 
paper  dealt  with  workload  as  a  primary,  secondary,  or  adjunct  topic.  If  the  topic  of  the 
paper  was  not  centered  on  workload  measurement  techniques  the  paper  was  dropped 
from  the  evaluation  process.  Each  document  which  contained  an  investigation  of 
workload  as  the  primary  topic  was  evaluated.  The  papers  were  evaluated  for  any 
investigation  of  the  validity  or  reliability  of  the  workload  measures  reported.  Each  paper 
received  a  second,  independent,  review  to  confirm  the  findings  of  the  first  reviewer. 

More  than  1400  titles  were  collected  in  the  Literature  Review.  Of  this  number  over  900 
titles  addressed  workload  as  either  a  primary  or  secondary  focus  of  study.  Of  that  set 
621  titles  addressed  the  workload  measures  function  and  factors  contained  in  FAR 
25.1523  Appendix  D.  Of  this  number,  319  titles  addressed  workload  measurement 
technique  reliability,  validity,  or  both.  In  final  form  the  fact  matrix  document  consists  of  an 
alphabetized  list  ot  all  titles  which  contained  workload  measurement  as  a  primary  topic  of 
empirical  investigation. 

The  following  is  a  list  of  the  workload  assessment  techniques  tabulated  in  the  Fact 
Matrices: 
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(a)  Subjective  Workload  Assessment  Technique  (SWAT) 

(b)  NASA  Task  Load  Index  (TLX) 

(c)  Workload  Compensation  Interference/Technical 
Effectiveness  scale  (WCI/TE) 

(d)  Modified  Cooper-Harper 

(e)  Interviews 

(f)  Surveys 

(g)  “Other"  subjective  measures 
PHYSIOLOGICAL 


(a) 

Body  Fluid 

(b) 

Brain  Activity 

(c) 

Heart 

d) 

Lung 

(e) 

Muscle 

(f) 

Skin 

(g) 

Vision 

(h) 

Voice 

0) 

"Other"  physiological 

PERFORMANCE 


(a)  Performance  Primary  Task:  Time 

(b)  Performance  Primary  Task:  Position 

(c)  Performance  Primary  Task:  Event 

(d)  Performance  Norma!  Secondary  Task:  Time 

(e)  Performance  Normal  Secondary  Task:  Position 

(f)  Performance  Normal  Secondary  Task:  Event 

(g)  Performance  Artificial  Secondary  Task:  Time 

(h)  Performance  Artificial  Secondary  Task:  Position 

(i)  Performance  Artificial  Secondary  Task:  Event 

The  Fact  Matrix  is  presented  as  Volume  2  of  Workshop  One  (Biferno  and  Boucek,  1987). 
The  fact  matrix  is  organized  into  three  sections.  First,  is  a  list  of  the  articles  contained  in 
the  fact  matrix  organized  alphabetically  by  first  authors  last  name.  Second,  is  a  list  of  all 
the  titles,  in  the  order  they  were  collected,  contained  in  the  fact  matrix  with  the 
corresponding  reference  number.  Finally,  the  cross  reference  of  measure,  by  Appendix 
D  function  or  factor,  by  reliability  and/or  validity  are  presented  in  the  fact  matrices.  The 
numbers  within  the  matrix  cells  (Table  3.2-1)  are  the  database  reference  numbers  which 
are  found  in:  "Proceedings  Of  The  Assessment  Of  Crew  Workload  Measurement 
Methods,  Techniques  And  Procedures:  Volume  II  -  Library  References." 


Example  of  Page  From  Fact  Matrix 

\  25  Workload  Factor  4a:  Degree  of  Mental  Effort 
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4.0  WORKSHOP  ONE  (LONG  BEACH,  CA) 

4.1  OBJECTIVES 


Workshop  number  one  was  conducted  in  order  to  obtain  agreement  among  workload 
experts  regarding  a  candidate  list  of  measures  for  simulation  testing  at  the  NASA-Ames 
Research  Center.  The  criteria  for  determining  the  candidate  workload  measures  was 
empirical  evidence  of  validity  and  reliability. 

4.2  ATTENDEES 

Fifty  workload  experts  attended  the  two-day  workshop  held  in  Long  Beach,  California. 
They  came  from  a  wide  cross  section  of  scientists  who  have  expertise  with  workload 
measurement.  The  area  of  workload  measurement  expertise  was  divided  into  the 
familiar  domains  of  subjective  ratings,  physiological  recordings,  and  performance 
,  measures. 

4.3  DESCRIPTION  OF  EVENTS 

On  the  first  day  presentations  were  made  by  experts  regarding  "state-of-the-art" 
'  workload  measures  broken  down  by  domain:  Subjective  Rating,  Physiological 

Measures,  and  Performance  Measures.  Three  experts  from  each  domain  made 
presentations  in  order  to  ensure  unbiased  representation  of  the  domain. 

On  the  second  day  the  group  was  divided  according  to  areas  of  expertise  for  further 
panel  discussions.  The  objective  of  the  panel  discussions  was  to  provide  an 
independent  review  of  the  facts  concerning  the  validity  and  reliability  of  workload 
measures.  The  attendees  were  asked  to  comment  on  the  results  of  the  literature  review 
and  participate  by  means  of  discussion.  A  free  exchange  of  ideas  was  encouraged  and 
documented.  The  questions  put  before  the  workshop  for  all  participants  to  answer  were: 

(a)  Given  the  workload  types  being  considered  for  measurement  in  a  transport 
environment,  which  measures  are  the  most  valid,  reliable,  and  applicable? 

(b)  On  what  evidence  do  you  base  your  opinion? 

Each  workshop  attendee  received  a  set  of  the  matrices,  one  matrix  for  each  of  the 
applicable  workload  types  mentioned  in  FAR  25.1523  Appendix  D.  The  matrices  were 
reviewed  and  modified  by  the  experts.  Each  workshop  participant  reviewed  and 
discussed: 

(a)  The  results  of  the  literature  review, 

(b)  The  criteria  which  was  suggested  for  selecting  candidate  workload 
measures, 

(c)  The  list  of  candidate  workload  measures  which  was  proposed  for  further 

♦  testing. 

Suggestions  were  solicited  for  additional  workload  assessment  techniques  (not  included 
in  the  matrices)  to  insure  that  all  measures  were  considered. 

A  two  volume  summary  of  the  proceedings  was  generated  from  the  first  workshop. 
Volume  One  contains  the  presentations  given  at  the  workshop  summarizing  the  current 
state  of  the  art  in  the  field  of  workload  assessment  (Biferno  and  Boucek,  1987a). 
Volume  Two  contains  a  bibliography  from  the  comprehensive  literature  review  that  was 
performed  and  the  fact  matrices  (Biferno  and  Boucek,  1987b). 
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4.4  RESULTS  OF  WORKSHOP 

The  final  list  of  candidate  measures  was  influenced  by  the  results  of  the  workshop  and 
constitute  the  workload  measures  which  were  considered  the  most  valid,  reliable,  and 
applicable  for  certification.  Workload  measures  suggested  from  the  panel  discussions, 
measures  identified  by  Douglas  and  Boeing  as  candidates  prior  to  Workshop  One,  and 
measures  actually  used  in  the  Part-Task  simulation  are  presented  below  (T able  4.4-1). 

Timeline  Analysis  was  used  as  an  analytic  tool  in  order  to  make  a  priori  predictions 
regarding  the  task  demands  imposed  on  the  crew.  For  a  measure  to  be  considered 
valid  it  must  be  able  to  discriminate  between  the  levels  of  low  and  high  workload 
operationalized  by  the  task  demands. 


* 


♦ 
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RECOMMENDATIONDS  OF  PARTICIPANTS  DOUGLAS/BOEING  TEAM  RECOMMENDATIONS  ACTUAL  MEASURES  USED  IN  SIMULATION  TEST 

AT  WORKSHOP  ONE 


Table  4.4-1 
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Sternberg  Task  (AUDITORY  &  VISUAL)  Sternberg  Task  (AUDITORY) 


5.0  DEVELOPMENT  OF  MISSION  SCENARIOS 

The  mission  scenarios  used  for  evaluating  a  new  aircraft  flightdeck  for  workload  requires 
careful  consideration  in  order  to  manipulate  task  demands  relevant  to  FAR  25.1523 
Appendix  D. 

During  the  aircraft  certification  process  the  manufacturer  provides  a  description  of  the 
flight  scenarios  for  the  simulation  testing,  flight  test,  or  the  "mini-airline"  operation,  for 
workload  evaluation  to  the  FAA.  The  level  of  detail  for  scenario  description  must  be 
sufficient  enough  to  allow  for  the  discussion  of  the  flightcrew  actions  that  are  to  be 
evaluated  in  flight. 

It  is  during  high  fidelity  simulation,  or  actual  operation,  that  workload  measures  provide 
information  about  the  workload  imposed  on  the  crew  in  order  to  confirm  that  they  can 
reliably  cope  when  in  airline  service. 

Seven  mission  scenarios  were  developed  for  the  simulation  testing  portion  of  this 
contract  effort.  Four  simulation  scenarios  were  used  in  the  Part-Task  simulation  testing: 
two  short  (30  minute)  segments,  a  high  and  a  low  workload  condition  of  each.  Three 
simulation  scenarios  were  used  in  the  Full-Mission  testing:  two  short  (30  minute) 
segments  and  one  long  (1:30  minute)  segments,  and  are  reported  in  the  Methods 
section  of  the  Full-Mission  simulation.  Detailed  descriptions  of  these  scenarios  can  be 
found  in  the  appropriate  simulation  design  sections;  7.1. 2.1  and  9.1. 2.1,  respectively. 

5.1  SELECTION  OF  CRITICAL  WORKLOAD  EVENTS 

A  sampling  of  normal  and  non-normal  procedures  was  implemented  in  the  simulation 
scenarios  to  manipulate  the  task  demands  of  the  flight  crew  in  order  to  vary  workload. 
The  face  validity  (realism)  of  the  task  demand  manipulations  was  verified  by  flight 
operations  personnel  (from  both  Douglas  and  Boeing).  Operating  considerations  such 
as  weather,  routes,  weight,  and  balance  were  considered  representative  of  actual  B-727 
operations.  Malfunction  conditions  were  selected  that  would  exercise  the  functions  and 
factors  of  FAR  25.1523  Appendix  D,  and  would  not  be  considered  improbable  events. 
Preplanned  dispatch-inoperative  items  (i.e.,  Autopilot  Inop)  that  could  result  in  added 
workload  were  incorporated  in  the  simulation  program. 

To  determine  that  the  high  workload  events  to  be  used  in  the  scenarios  demonstrated 
face  validity  there  was  close  coordination  with  flight  operations  personnel  (from  Douglas 
and  Boeing)  to  ensure  the  following:  malfunctions  occurred  in  the  scenarios  at  a  logical 
time  and  phase  of  flight  and  required  operationally  correct  responses  from  the  crew; 
multiple  failures  occurred  in  a  logical  order  and  were  representative  of  malfunctions 
experienced  in  airline  operations.  High  workload  events  were  also  coordinated  in  the  Air 
Traffic  Control  (ATC)  scripts  with  ATC  personnel. 

The  method  of  "triggering"  the  malfunction  was  planned  so  that  the  malfunction  occurred 
consistently  across  evaluation  flights. 

5.2  SCENARIO  DESCRIPTION 

The  following  is  a  list  of  items  essential  for  thorough  scenario  description.  All  the 
following  items  should  be  considered  in  the  development  of  scenarios  used  in  a 
certification  effort. 


18 


5.2.1  WEATHER 


A  consideration  for  simulation  testing  is  the  weather  conditions  to  be  experienced  during 
the  evaluation.  The  weather  conditions  for  the  scenario  should  include  both  Instrument 
Meteorological  Conditions  (IMC)  and  Visual  Meteorological  Conditions  (VMC).  In 
addition,  ammeter  setting,  winds,  temperature,  ceiling,  visibility,  and  dewpoint  need  to  be 
specified.  The  weather  can  then  be  developed  for  each  of  the  airfields  used  in  the 
scenario.  Winds  aloft,  surface  observations  (ATIS),  terminal  forecasts  should  all  be 
developed  for  the  simulation  test. 

5.2.2  FLIGHT  ENVIRONMENT 

CLEARANCES  AND  ROUTING 

As  with  all  other  aspects  of  flight  operations,  there  must  be  face  validity  for  the  scenarios 
to  be  flown  for  workload  evaluation.  As  departure  and  landing  charts  are  periodically 
updated,  either  the  simulation  scenario  must  be  revised  for  accuracy,  or  the  scenario 
must  be  frozen  to  Standard  Instrument  Departures  (SID)  &  Standard  Terminal  Arrival 
Routes  (STAR)  of  specific  dates,  which  are  tnen  supplied  with  the  flight  briefing  material. 
The  latter  approach  was  used  in  this  study. 

Inoperative  navigation  aids,  changes  in  typical'  runways  at  an  airport,  diversions  owing 
to  weather,  and  missed  approaches  are  all  appropriate  considerations  for  manipulations 
of  workload  for  the  purpose  of  simulation  scenario  specification. 

FLIGHT  PLANS 

Flight  plans  should  be  provided  for  each  route  simulated.  The  flight  plan  includes 
distance  to  each  waypoint,  time  between  waypoints,  and  total  elapsed  time  along  the 
route.  Flight  plans  must  take  into  account  required  climb  and  cruise  airspeeds,  and  ATC 
requested  clearances  and  routes.  From  the  flight  plan  a  mission  summary  can  be 
developed  that  includes  more  detailed  events  such  as  when  the  malfunctions  occur. 

DISPATCH  RELEASE  PACKAGES 

Before  flight,  subject  pilots  should  be  given  a  dispatch  release  package  which  includes 
data  representative  of  the  information  provided  for  a  revenue  airline  flight.  This  includes 
a  skeleton  flight  plan,  route  data,  weather  information,  weight,  and  balance  data,  Notice 
to  Airmen  (NOTAMS),  Convective  Weather  Information  (SIGMETS),  and  known 
inoperative  items  complying  with  the  Minimum  Equipment  Ust  (MEL).  The  dispatch 
release  packets  used  in  the  simulation  studies  are  included  in  Appendix  A. 

AIR  TRAFFIC  CONTROL  (ATC^  SCRIPTS 

Compliance,  as  well  as  attending  to  ATC  direction  is  a  significant  source  of  workload  for 
the  commercial  transport  environment.  ATC  interaction  with  the  flight  crew  should  be 
incorporated  in  an  evaluation  of  crew  workload. 

Included  with  the  ATC  script  should  be  a  list  of  frequencies  used  during  the  scenarios. 
Included  in  the  list  of  frequencies  are  dispatch,  clearance  delivery,  departure  control, 
ATIS,  tower  (at  the  departure  and  arrival  airports),  enroute  centers  and  approach  control 
frequencies.  The  radio  frequencies  should  be  included  in  the  dispatch  release 
packages. 
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The  ATC  scripts  used  in  the  Full-Mission  simulation  testing  performed  at  NASA-Ames 
(Appendix  C)  provide  an  example  of  the  detail  that  was  required  for  ATC  simulation  in  the 
simulation  testing  environment. 

The  capability  exists  for  full  ATC  support  at  the  NASA  Ames  727  simulator  facility; 
therefore,  ATC  scripts  had  to  be  developed.  ATC  communications  required  by  the 
scenario  were  integrated  into  the  detailed  scenario  that  was  built. 

During  the  simulation  tests  ATC  communication,  similar  to  revenue  service,  included 
many  calls  to  'other'  aircraft  in  the  ATC  system  in  order  to  replicate  "real  world" 
conditions. 

In  addition,  the  visual  system  was  programmed  to  support  a  percentage  of  the  'other' 
aircraft  as  visual  targets  in  order  to  enhance  simulation  fidelity.  The  remainder  of  the 
ATC  calls  to  'other*  aircraft  did  not  include  visual  targets.  To  ensure  that  the  ATC  calls 
occurred  at  the  same  point  in  each  flight  during  the  simulation  tests  was  a  difficult  task. 
The  ATC  personnel  were  provided  operational  cues  (i.e.,  measurement  window 
openings  and  closings)  to  aid  in  their  attempt  to  make  calls  in  a  consistent  fashion.  The 
controllers  used  lights  associated  with  data  window  'triggers'  to  aid  in  the  timinq  of  their 
interaction. 

5.2.3  AIRCRAFT  CONSIDERATIONS 
WEIGHT  AND  BALANCE 

Fuel  and  passenger  loading  calculations  should  be  computed  for  each  route  including: 
gross  weight  at  takeoff,  fuel  at  takeoff,  the  amount  of  fuel  burned,  fuel  remaining  after 
landing,  and  the  landing  gross  weight.  This  information  affects  takeoff  and  landing 
speeds,  required  runway  lengths,  and  all  aircraft  performance  characteristics. 

PHASES  OF  FLIGHT 

The  phases  of  flight  for  the  scenario  should  be  specifically  defined  for  data  collection 
purposes,  such  as  those  described  in  the  simulation  design  sections  7.1. 2.1  and 
9. 1.2.1.  The  phases  of  flight  used  in  the  following  simulation  scenarios  were  of  varying 
length,  but  incorporated  the  operational  procedures  associated  with  takeoff,  climb,  top 
of  climb,  cruise,  top  of  descent,  approach,  and  landing. 

MALFUNCTIONS  AND  MINIMUM  EQUIPMENT  LIST  (MEL)  ITEMS 

Malfunctions  should  be  planned  and  coordinated  so  that  they  occur  logically  in  the 
scenario  and  require  operationally  correct  responses  from  the  crew.  The  method  of 
triggering'  a  malfunction  should  be  planned  so  the  malfunction  occurs  consistently 
across  the  subject  population.  Pilot  deviation  from  course  or  altitude  must  be 
considered  when  specification  of  the  malfunction  is  considered. 

Dispatching  the  aircraft  with  allowable  systems  inoperative  is  another  technique  for 
manipulating  workload.  Autopilots  or  primary  pressurization  systems  inoperative  are 
candidate  manipulations  to  vary  task  demands. 

DATA  COLLECTION  WINDOWS 

When  collecting  data  for  specific  phases  of  flight  during  the  workload  evaluation 
scenario,  defining  the  data  collection  periods  (i.e.,  data  windows)  is  very  important.  The 
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method  of  triggering'  the  opening  and  closing  of  the  windows  must  be  chosen  to  cause 
the  same  events  to  occur  in  the  same  measurement  window  across  the  subject 
population.  The  windows  should  be  triggered  whenever  possible  by  system  defined 
events  such  as  "flaps  up"  or  the  aircraft  crossing  a  certain  prescribed  altitude  during  a 
profile  descent.  If  operational  events  cannot  be  determined  so  as  to  preclude  individual 
variation  from  flying  performance,  windows  can  be  triggered  by  time  events;  for  instance, 
a  window  can  be  closed  2  minutes  after  it  opens.  If  a  malfunction  is  planned,  the  window 
should  be  triggered  to  open  prior  to  the  malfunction. 

DISTRACTION  SCENARIOS 

In  addition  to  malfunctions,  distractions  were  included  in  scenarios.  When  the  same  pilot 
flew  the  same  scenario  in  test  and  retest,  these  distractors  helped  disguise  the  fact  that 
the  same  scenario  was  being  repeated.  Examples  of  distractions  used  are  shown 
below: 


AUTO  PRESSURE  FAILURE  (AFTER  TAKEOFF,  PASSING  4000  FT) 

*F/E  REPORTS,  “AUTO  PRESSURE  CONTROLLER  JUST  FAILED. 

SWITCHING  TO  STANDBY." 

*CAPT  SAYS,  "ROGER,  IS  STANDBY  WORKING  OK?" 

*F/E  SAYS,  "STANDBY  LOOKS  GOOD.  WE'LL  OPERATE  IN 
STANDBY  THE  REST  OF  THE  TRIP." 

*CAPT  ACKNOWLEDGES. 

PACK  TRIP  (AFTER  TAKEOFF  PASSING  3000  OR  DESCENT  PASSING  5000FT) 

*F/E  REPORTS,  "WE  HAVE  A  PACK  TRIP  ON  THE  RIGHT  PACK." 

*CAPT  SAYS,  "ROGER,  SELECT  A  WARMER  TEMPERATURE  AND  TRY 
A  RE-SET " 

*F/E  ACCOMPLISHES  PACK  TRIP  OFF  CHECKLIST  AND  SAYS, 

"PACK  RESET  OK,  CHECKLIST  COMPLETE." 

*CAPT  ACKNOWLEDGES. 

ELEVATOR  FEEL  DIFFERENTIAL  LIGHT 

*F/E  REPORTS,  "CAPT.,  THE  ELEVATOR  FEEL  DIFFERENTIAL 
LIGHT  IS  ILLUMINATED.  hYDRAULlC  PRESSURES  AND  QUANTITIES 
OK.  CHECKLIST  SAYS  TO  FLY  NORMALLY  AND  AVOID  ABRUPT 
ELEVATOR  INPUTS. 

*CAPT  ACKNOWLEDGES. 

WINDOW  OVERHEAT  (AFTER  TAKEOFF  PASSING  4000  FEET  OR  DURING  DESCENT 

PASSING  9000  FEET) 

*F/E  REPORTS,  "CAPT.,  WE  HAVE  AN  OVERHEAT  ON  THE  R-1 
WINDOW.  I’LL  TURN  OFF  THE  SWITCH  AND  LET  IT  COOL, 

THEN  WILL  TRY  A  RE-SET. 

*CAPT  ACKNOWLEDGES. 

*CAPT  NOTES  F/E  TURN  OFF  R-1  WINDOW  HEAT  SWITCH. 

*F/E  REPORTS,  "UNABLE  TO  RE-SET.  CHECKLIST  SAYS 
TO  LIMIT  AIRSPEED  TO  200  FEET  BELOW  10,000  FEET, 

AND  TURN  WINDOW  DE-FOGGER  ON." 
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*CAPT  ACKNOWLEDGES  AND  NOTES  F/O  TURNING  DE-FOGGER 
ON. 


These  distractions  did  not  occur  during  data  collection  windows. 

5.3  PROCEDURE  USED  TO  CATEGORIZE  OPERATIONALLY  RELEVANT 
WORKLOAD  WITH  FAR  25.1523  Appendix  D 

To  ensure  the  functions  and  factors  of  FAR  25.1523  Appendix  D,  are  represented  by  the 
task  demands  used  in  testing  a  scheme  is  developed  for  mapping  the  functions  and 
factors  onto  the  tasks  required  by  the  mission  scenarios.  The  goal  is  to  evaluate  the 
specific  workload  types  (FAR  25.1523  Appendix  D,  functions  and  factors)  with  the 
appropriate  crew  complement  during  realistic  operating  conditions,  including 
representative  weather,  air  traffic,  and  airline  operational  duties. 

The  scenarios  built  for  the  simulation  tests  were  based  on  the  incorporation  of 
operationally  relevant  types  of  task-demands  being  placed  on  the  flightcrew.  Flight 
scenarios  were  specified  to  include  task-demands  which  pose  a  concern  for  transport 
flightcrew  workload.  An  analytical  assessment  of  these  task-demands  was 
accomplished  using  the  Boeing  Commercial  Airplanes  Time-Line  Analysis  (TLA). 

Workload  is  thought  to  be  a  multidimensional  construct  combining  the  demands 
imposed  on  the  pilot  as  he  attempts  to  achieve  the  flight  objectives,  and  the  momentary 
capacity  of  the  pilot  to  meet  these  demands.  It  is  important  that  the  workload  evaluation 
scenario  simulate  a  multi-faceted  environment  to  provide  a  representative  range  of  task 
demands  for  the  subject  pilots.  While  the  tasks  performed  by  the  subjects  in  evaluation 
studies  should  be  representative  of  those  performed  in  actual  flight  operations,  care 
should  be  taken  to  map  the  workload  experienced  in  performing  the  tasks  to  the 
functions  and  factors  of  FAR  25.1523  Appendix  D. . 

The  task  of  operationally  defining  the  functions  and  factors  of  Appendix  D  should 
include:  Specification  of  the  mission  segments  to  be  included  in  the  evaluation,  and 
development  and  implementation  of  a  categorization  scheme  to  map  the  specific  tasks 
of  the  flightcrew  to  the  functions  and  factors  found  in  FAR  25.1523  Appendix  D.  A 
description  of  the  process  of  mapping  functions  and  factors  to  specific  tasks  follows. 

Scenarios  were  constructed  which  described  the  flightcrew's  actions  down  to  the  task 
and  subtask  level.  A  representative  task  is  "tune  VHF  radio  1  to  123.9."  Subtask 
description  goes  a  step  further  such  as,  "move  right  hand  to  VHF  radio  1  channel 
selector  knob",  "turn  knob  counter-clockwise  to  123.9,"  "confirm  frequency  123.9  is 
visible  in  display  window,"  "return  right  hand  to  rest." 

Each  workload  function  and  workload  factor  of  FAR  25.1523  Appendix  D,  was  assigned 
an  operational  definition.  The  operational  definitions  that  were  generated  and  used  in 
this  study  are  shown  in  Figures  5.3-1  through  5.3-4.  Using  these  definitions  assured 
consistency  when  mapping  the  functions/factors  onto  the  tasks  of  the  scenarios.  The 
creation  of  the  operational  definitions  included  discussions  with  flight  operations 
personnel  from  Douglas  and  Boeing.  These  rules  could  be  defined  differently  with  little 
consequence  as  long  as  they  were  used  consistently  in  the  mapping  process.  In  fact, 
when  two  experimenters  (one  Douglas  and  one  Boeing)  independently  assigned 
function /factor  mappings  to  the  steps  of  the  scenarios  using  the  pre-established 
definitions,  their  mappings  were  nearly  identical.  For  the  few  differences  that  did  exist, 
flight  operations  personnel  were  consulted  and  the  differences  reconciled. 
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Functions 


Figure  5.3-1 
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Figure  5.3-2 
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1.  Controls 


Figure  5.3-3 
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10.  Crew  member  incapacitated 


An  example  of  the  function  and  factor  mapping  is  shown  in  Figure  5.3-5.  As  shown,  the 
functions  and  factors  are  not  mutually  exclusive.  In  fact,  many  steps  of  the  scenarios 
represent  three  or  more  functions  and  three  or  more  factors.  By  the  pre-defined  rules, 
the  action,  "calls,  'gear  up1",  for  example,  represents  functions  4,  5,  and  6 
(communications,  operations  and  monitoring  of  aircraft  engines  and  systems,  and 
command  decisions,  respectively)  and  factors  3  and  8A  (procedures  and 
communication,  respectively).  The  level  of  task  specification  done  for  the  scenarios  can 
also  be  seen  in  the  task  listing  included  in  Appendix  B.  In  a  20  second  period  of  time,  six 
independent  tasks  are  performed,  each  representing  a  variety  of  functions  and  factors. 

The  mapping  procedure  assured  that  a  priori  predictions  of  workload  differences  could 
be  made  prior  to  collecting  data  using  the  workload  assessment  techniques  in  the 
simulation  at  NASA-Ames.  Measurement  periods  within  the  flight  could  be  identified 
where  workload  levels  were  high  or  low.  Function  and  factor  tally  sheets  were  prepared 
for  each  of  the  scenarios  indicating  frequency  counts  for  the  number  of  occurrences  of 
each  function  and  factor.  An  example  tally  sheet  is  shown  in  Figure  5.3-6. 

To  operationalize  the  differences  between  mental  and  physical  workload,  each  function 
and  factor  was  assigned  to  either  mental  or  physical.  No  task  is  purely  physical  or  purely 
mental;  however,  the  assumption  was  made  that  tasks  do  lie  along  a  continuum  from 
pnysical  to  mental  workload,  and  therefore,  the  task  could  be  assigned  to  the  end  of  the 
continuum  that  more  closely  described  it.  This  procedure  allowed  physical  and  mental 
task  loadings  to  be  made  for  each  phase  of  flight,  for  evei7  scenario.  Mental  workload 
increases  with  mediational  tasks  such  as  perception,  cognitive  processing,  and  decision 
making.  Physical  workload  increases  with  gross  motor  movement.  In  this  manner,  it 
could  be  judged  which  measurement  periods  contained  relatively  high  levels  of  mental 
workload  and  which  involved  predominantly  physical  workload. 
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Function  and  Factor  Mapping 

Example 


Figure  5.3-5 
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|  pitch  attitude 

00:02:35  Cleared  Direct  Sacramento  Vortac 


Function  and  Factor  Tally  Sheet 


Figure  5.3-6 
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9(5)  Monitoring 
9(8A)  Communication 
9(8B)  Navigation 


6.0  ROLE  OF  TIME-LINE  ANALYSIS 


The  analytical  measure  used  to  predict,  a  priori,  levels  of  workload  was  Timeline  Analysis 
(TLA).  TLA  computes  the  ratio  of  time  required,  that  is,  execution  time,  to  time  available 
throughout  a  mission  scenario  (Miller,  1976).  A  serious  criticism  of  TLA  is  the  serial 
approach  it  takes  in  calculating  task  execution  when  it  is  known  that  pilots  can  conduct 
multiple  actions  in  parallel.  Flaws  in  the  method  leads  to  an  over-estimation  of  workload, 
an  error  on  the  side  of  safety  which  is  appropriate  when  evaluating  a  new  aircraft. 
Boeing  Commercial  Airplane's  TLA  technique  was  used  in  this  study  because  of  the 
database  available  for  the  B-727  aircraft  which  was  the  testbed  for  the  simulation  studies. 

6.1  OBJECTIVE 

The  timeline  analysis  was  used  to  identify  high  and  low  task  demand  levels.  A  detailed 
task  timeline  analysis  was  performed  on  the  flight  scenarios  to  be  used  in  the  simulation 
testing  at  NASA-Ames. 

6.2  DESCRIPTION  OF  TLA  TASK 

Timeline  analysis  uses  micro-motion  analysis  techniques  to  compute  a  ratio  of  time 
required  to  complete  tasks  to  the  time  available.  Timeline  analysis  breaks  down  the  ratio 
of  time  required  to  time  available  for  different  body  channels  (i.e.,  vision,  manual  left  & 
right,  auditory,  cognitive,  and  verbal).  To  support  the  Boeing  Timeline  Analysis  program 
it  was  necessary  to  build  a  geometric  data  base  describing  the  flight  deck  of  the  NASA- 
Ames  727  simulator.  The  simulator  is  a  Delta  configuration  727  series  232.  The 
geometric  data  base  file  contains  control  and  instrument  descriptions  and  locations  and 
related  information.  It  contains  airplane  coordinate  locations  of  the  flightdeck  instrument 
panels  and  the  rectangular  coordinates  of  the  controls  and  indicators  on  each  panel. 
These  coordinate  locations  are  based  on  engineering  drawings  of  the  cockpit  and 
allows  the  flight  deck  to  be  evaluated  from  any  angle  (Figures  6.2-1,  6.2-2,  and  6.2-3). 

Time  based  mission  scenarios  were  built  describing  the  crew  activity  associated  with 
flying  the  routes  used  during  testing  at  NASA  Ames.  Creation  of  the  scenario  is  based 
upon  data  derived  from  mission  flight  plans,  maps,  approach  and  landing  charts, 
interaction  with  ATC,  aircraft  performance  data  and  aircraft  operations  manuals.  Using 
this  information  base,  detailed  procedures  were  developed  which  defined  the  actions  a 
crew  member  must  accomplish  to  successfully  complete  a  mission  (Figures  6.2-4). 
These  procedures  establish  the  basic  work  time  units  from  which  the  TLA  workload 
statistics  were  derived.  Normal  as  well  as  high  workload  procedures  were  developed  for 
the  routes. 

TLA  represents  workload  requirements  for  various  body  channels  including:  Visual, 
manual  (left,  right,  and  both),  verbal,  auditory  and  cognitive.  Execution  time  estimates 
are  calculated  in  terms  of  hand  and  eye  motions  used  to  execute  a  procedure  and  the 
time  in  transit  required  to  accomplish  these  motions  (Figure  6.2-5).  Manual  motion  is 
calculated  from  one  control  to  the  next  along  a  curve  to  simulate  lifting  the  hand  from 
one  control,  moving  it  in  an  arc  to  the  next  control  and  lowering  the  hand.  Dwell  times 
required  to  use  a  specific  control  or  display  are  selected  from  stored  data  tables. 
Summing  the  transit  and  dwell  times  for  each  action  produces  an  estimate  of  the  total 
time  required  by  an  operator  to  successfully  complete  the  procedure. 

In  the  geometric  data  base  each  device  on  the  flight  deck  is  characterized  in  terms  of  it's 
location,  dwell  time,  and  complexity  score  (Figure  6.2-6).  This  device  complexity  score  is 
based  upon  the  information  content  of  the  possible  states  the  device  presents  (Figure 
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Flight  Deck  Geometric  Data  Base 


Figure  6.2-1 
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Timeline  Analysis 
Flight  Deck  Geometric  Data  Base 


Figure  6.2-2 
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Timeline  Analysis 

Example  of  Flight  Mission  Scenario  File 
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Figure  6.2-3 
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Figure  6.2-4 
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Checks  altimater  for  positive  rate  of  climb 
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Timeline  Analysis 

Time  Requirements 
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Horizontal  Positioning  Movements  Eye  Motion  Versus 

Versus  Length  of  Movement  Angle-of-Eye-Movement  Change 


Timeline  Analysis 

Crew  Station  Description,  Example 
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Figure  6.2-6 
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6.2-7).  It  serves  as  the  basis  for  estimating  cognitive  workload.  The  procedure 
complexity  is  the  sum  of  the  device  complexity  scores  for  all  steps  of  the  procedure. 

Cognitive  workload  is  computed  by  an  algorithm  that  relates  the  complexity  score  to  the 
number  of  choices.  The  algorithm  calculates  how  long  it  takes  a  person  to  act 
depending  on  the  number  of  choices  available.  Verbal  and  auditory  response  times 
were  derived  from  data  contained  in  the  Index  of  Electronic  Equipment  Operability. 

Workload  estimates  are  made  for  each  task,  procedure,  phase  of  flight,  and  for  the  entire 
scenario.  Analyses  were  completed  only  for  the  captain's  tasks. 
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Timeline  Analysis 

Device  Complexity  Measure 


Figure  6.2-7 
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7.0  PART-TASK  SIMULATION  TESTING 

The  Part-Task  simulation  was  performed  at  the  NASA-Ames  Research  Center,  Moffett 
Field,  California.  To  ensure  the  generalizability  of  the  test  results  to  the  real  world,  the 
Part-Task  testing  environment  was  as  similar  to  the  operational  environment  as  possible 
(e.g.,  the  flight  deck  of  the  production  aircraft  during  certification  flights). 

7.1  METHOD 

Only  Captains  were  used  for  the  following  evaluation.  Newer  aircraft,  with  two 
crew  flightdecks,  do  not  have  the  wide  variety  of  tasks  found  in  older  flightdecks.  The 
newer  generation,  two  crew  aircraft,  normally  divide  the  responsibilities  so  that  one  crew 
member  is  considered  the  Pilot  Flying  (PF)  and  the  other  is  the  Pilot  Not  Flying  (PNF). 
The  PF  controls  the  aircraft,  either  manually  or  using  the  autopilot,  has  primary 
responsibility  of  the  navigation  of  the  aircraft,  and  monitors  the  aircraft  systems.  The 
PNF  handles  communications,  is  the  primary  system  monitor,  and  serves  as  a  backup 
for  the  navigation  process.  The  Captain  and  First  Officer  exchange  these  roles  readily. 
For  this  reason  it  was  deemed  sufficient  to  instrument  and  record  workload  from  only  the 
pilot  in  the  left  seat  in  the  following  study.  No  attempt  was  made  to  address  crew 
performance  when  PF  and  PNF  duties  are  exchanged,  for  a  thorough  discussion  see 
Orlady  (1982). 

7.1.1  SUBJECTS 

PILOTS 

Eighteen  Airline  Transport  Pilots  (ATP),  (from  American,  United,  Delta,  TWA,  and 
Eastern)  served  as  subjects  in  the  experiment.  Subjects  were  all  male  ranging  in  age 
from  44  to  58.  Subjects  were  either  currently  F.A.R.  Part  121  qualified  as  Captain  for  the 
B-727  or  had  spent  5  years  of  duty  as  Captain  for  the  B-727. 

CONFEDERATE  FLIGHT  CREW 

Two  confederates  participated  in  the  simulation  study  as  the  First  Officer  and  Flight 
Engineer.  Preston  Suit,  a  member  of  Flight  Crew  Training  at  Boeing  Commercial 
Airplanes,  served  as  the  First  Officer  in  the  study.  Preston  also  gave  the  briefing  for 
differences  training  and  the  routes  to  the  pilots  participating  in  the  study.  Michael 
Bortolussi,  Western  Aerospace  Inc.,  served  as  the  Flight  Engineer  in  the  study.  Both  the 
First  Officer  and  Flight  Engineer  were  cognizant  of  the  workload  manipulations,  and 
attempted  to  give  each  pilot  similar  treatment  during  the  simulation. 

7.1.2  EXPERIMENTAL  DESIGN 

The  factors  that  drove  the  design  included: 

(a)  Different  levels  of  workload  as  defined  by  the  functions  and  factors  of  FAR 
25.1523  Appendix  D, 

(b)  Two  test  sessions  in  order  to  evaluate  reliability, 

(c)  Sampling  of  various  phases  of  flight  in  order  to  represent  task  demands 
representative  of  operational  conditions. 
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7.1. 2.1  INDEPENDENT  VARIABLE 

TEST/RETEST 

The  method  employed  in  the  present  study  utilizes  Test/Retest  as  a  means  of 
determining  the  reliability  of  the  various  workload  measures.  This  meant  that  the  pilots 
involved  in  the  study  participated  on  two  separate  occasions.  The  period  between  the 
two  simulation  test  periods  was  at  least  10  days,  and  was  as  high  as  42  days  in  one 
case. 

LOW  AND  HIGH  WORKLOAD  LEVELS 

There  were  two  different  levels  of  workload:  low  and  high.  The  low  workload  flight  is  a 
'  nominal'  flight,  no  equipment  is  MEL,  the  weather  is  clear  with  light  winds.  The  high 
workload  flight  contains  Instrument  Meteorological  Conditions  (IMC)  and  the  winds  aloft 
are  stronger.  Also  in  the  high  workload  flight  malfunctions  are  encountered.  At  the  top 
►  of  the  climb  segment  the  Number  three  engine  fails,  and  3  minutes  later  the  ”B"  system 

hydraulic  system  loses  quantity  and  pressure  resulting  in  a  total  failure.  The  autopilot  is 
INOP  in  the  high  workload  flight  as  well. 

A  table  is  provided  that  contains  a  summary  of  the  workload  manipulations  in  order  to 
aid  the  reader  in  understanding  the  different  task  demands  for  the  low  and  high  workload 
flights  (Table  7. 1.2. 1-1). 

The  manipulation  of  workload,  low  and  high,  and  flight  routes,  San  Francisco  to  Stockton 
and  Sacramento  to  San  Francisco,  were  counter-balanced  across  subjects. 

PHASES  OF  FLIGHT 

Seven  phases  of  flight  were  examined  in  the  simulation  test: 

(a)  Takeoff 

(b)  Climb 

(c)  Top  of  Climb  (TOC) 

(d)  Cruise 

(e)  Top  of  Descent  (TOD) 

(f)  Approach 

(g)  Landing 

Each  flight  contained  seven  measurement  "windows"  to  assess  the  seven  different 
*  phases  of  flight.  The  term  window  is  used  to  give  the  idea  of  a  momentary  examination 

of  a  portion  of  a  well  defined  phase  of  flight.  Window  and  phase  of  flight  are  used 
synonymously.  Window  is  used  when  referring  to  experimental  design  or  measurement 
period,  while  phase  of  flight  is  used  when  discussing  results.  The  events  which  opened 
and  closed  the  windows  is  listed  below: 
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Two  Workload  Levels 


Table  7.1 .2.1-1 
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failure,  window  overheat) 


PHASE  OF  FLIGHT 

OPENING  EVENT 

CLOSING  EVENT 

(a)  TAKEOFF 

E.P.R.  >  1.5 

Flaps  5  degrees 

(b)  CLIMB 

Flaps  up 

1  Minute  later 

(c)  TOP  OF  CLIMB 

10,000  feet 

2  Minutes  later 

(d)  CRUISE 

3  Minutes  after 

10,000  feet 

21/2  Minutes 
later 

(e)  TOP  OF  DESCENT 

Throttles  to  idle 

5,500  feet 

(0  APPROACH 

Localizer 

Activation 

Outer  Marker 

(g)  TOUCHDOWN 

Middle  Marker 

11/2  Minutes 
later 

A  graphic  is  provided  to  illustrate  the  flight  scenarios  pictorially  in  order  to  aid  the  reader 
in  understanding  the  measurement  windows  (Figure  7.1. 2.1-1). 

7.1. 2.2  DEPENDENT  VARIABLE 

SUBJECTIVE  RATING 

The  Subjective  Workload  Assessment  Technique  (SWAT),  NASA-Task  Load  Index 
(NASA-TLX),  and  a  simple  1-to-20  point  overall  workload  score  were  used  for  subjective 
workload  assessment. 


Half  of  the  subjects  used  the  Subjective  Workload  Assessment  Technique  (SWAT) 
(Figures  7.1. 2.2-1  and  7.1. 2.2-2)  and  the  other  half  used  the  NASA  Task  Load  Index 
(TLX)  (Figures  7. 1.2. 2-3  and  7. 1.2. 2-4)  with  the  1-to-20  point  overall  workload  scale 
appended  to  the  bottom  of  the  NASA-TLX  rating  page. 

In  the  current  paradigm,  the  implementation  of  the  subjective  rating  technique  was  not 
possible  in-flight  due  to  the  contamination  of  the  other  measures.  Owing  to  the 
recording  of  control  input  activity  it  was  decided  to  be  inappropriate  to  hand  the  pilot  a 
paper  and  pencil  based  subjective  rating  technique.  To  assess  the  subjective  workload 
tools  (SWAT  and  NASA-TLX)  half  of  the  subjects  received  SWAT,  the  other  half  received 
the  NASA-TLX,  during  post-flight  videotape  viewing.  An  attempt  made  to  gather 
subjective  ratings  closer  in  time  to  the  actual  flights.  Shortly  after  landing  the  pilots  were 
handed  a  clipboard  with  four  segments  to  the  flight  demarcated:  (1)  takeoff  through  top 
of  climb,  (2)  Cruise,  (3)  top  of  descent  through  approach  and  landing,  and  (4)  an  overall 
rating  for  the  entire  flight.  Those  results  are  discussed  in  another  publication  (Battiste 
and  Bortolussi,  1988). 

Both  SWAT  and  the  NASA-TLX  require  techniques  to  customize  the  event  ratings  so  they 
can  be  combined  to  yield  a  single  0-to-100  scale  for  each  measurement  window. 
Utilizing  conjoint  measurement  techniques  the  0-to-100  score  for  SWAT  is  based  on  an 
interval  scale.  The  technique  used  to  customize  the  NASA-TLX  0-to-100  score  does  not 
yield  a  truly  interval  0-to-100  scale  in  a  statistical  sense,  but  it  will  be  treated  as  such  in 
the  analyses. 


Phases  of  Flight  and  Data  Collection 


F i gure  7 . 1 . 2 . 1 -1 
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Passes  Passes 

5,500  ft  Outer 

Marker  aliwos 


Figure  7.1 .2.2-1 

SWAT  Rating  Scale 

Subfact  No _ _ 

Date _ 

Mission  Sagmant _ 

I.  Tana  load 

_ 1.  Oban  hava  apart  tana.  Interruptions  or  ower- 

lap  among  activities  occur  Mraquantfy  or  not  at  at. 

_ 2.  Oocationatty  hava  tewra  Mma.  Intomjtiions  or 

overlap  among  adMtias  occur  tiaquantty. 

—  3.  Almost  never  hava  spam  Nma.  Interruptions  or 
overlap  among  activities  art  vary  Frequent  or  occur 
at  tra  lima. 


L  Martial  aflort  load 

—  t.  Vary  title  conscious  mantel  aflort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  fittfe  or 
no  attention. 

_ 2.  Moderate  conscious  martial  effort  or  concentration 

required  Complexity  or  activity  is  moderately  high 
due  lo  uncertainty,  unpredfctabtity,  or  untamilartty 
Considerabie  attention  required. 

_ 3.  Extensive  mental  effort  and  concentration  are  necessary. 

Very  complex  activity  requiring  total  attention. 

M.  Psychological  stress  load 

— 1.  Littte  confusion,  risk,  frustration,  or  anxiety 
exists  and  can  be  easiy  accommodated. 

_ 2.  Moderate  stress  due  to  confusion,  frustration,  or 

anxiety  noticeably  adds  lo  wortdoad.  Significant 
compensation  b  required  to  maintain  adequate  performance. 

— 3  Kfigh  to  vary  Intense  stress  due  to  contusion, 
frustration,  or  anxiety.  High  to  extreme  determination 
and  self  oontrol  requved. 

F 1 167 .34A  R2db 


SWAT 

27  Different  Combinations 


Time  load 


Medium  (2) 


Low  (1) 


Mental  effort 


^  High  (3) 

Medium  (2) 
Low  (1) 


High  (3) 

Lo“^S.ress  load 


[High  (3) 
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Figure  7. 1.2. 2-3 


TLX  Rating  Scale  Definitions 

Title  Endpoints  Descriptions 


Mental 

Demand 

Low/High 

How  much  mental  and  perceptual  activity  was  required 
(e.g.  thinking,  deciding,  calculating,  remembering, 
looking,  searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving? 

Physical 

Demand 

Low/High 

How  much  physical  activity  was  required  (e.g.,  pushing, 
pulling,  turning,  controlling,  activating,  etc.)?  Was  the 
task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous, 
restful  or  laborious? 

Temporal 

Demand 

Low/High 

How  much  time  pressure  did  you  feel  due  to  the  rate 
or  pace  at  which  the  tasks  or  task  elements  occurred? 

Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

Performance 

Perfect/Failure 

How  successful  do  you  think  you  were  in  accomplishing 
the  goals  of  the  task  set  by  the  experimenter  (or  yourself)? 
How  satisfied  were  you  wi'h  your  performance  in 
accomplishing  these  goals? 

Effort 

Low/High 

How  hard  did  you  have  to  work  (mentally  and  physically) 
to  accomplish  your  level  of  performance? 

Frustration 

Level 

Low/High 

How  insecure,  discouraged,  irritated,  stressed  and 
annoyed  versus  secure,  gratified,  content,  relaxed, 
and  complacent  did  you  feel  during  the  task? 

’  3  F1 167.32  R3G 

Figure  7.1 .2.2-4 

NASA  TLX  Rating  Scale 

Subject  ID: _ Task  ID: _ 


Rating  Sheet 

Mental  demand 

.1.1.1. 


low 

Physical  demand 
l_i _ 1 _ ■ _ t  i. 


Low 

Temporal  demand 


Low 

Performance 


_i _ L 


Good 

Effort 


_i _ i _ i _ i _ i_ 


I  .  I  .  1  .  I  . 


Low 

Frustration 
I  ,  I 


Low 


I  ,  I 


Overall  workload 

.1,1. 


I  Low 


-I - 1 _ I - 1 - L. 


I  ■  l  ■  I 


Li  I 


High 


■  I  .  1  -a 


High 


I  .  I  .  I 


High 


1  .  I  .  1  .  I  . 


Poor 


■j-JL-i. 


High 


High 


. . L — i — J _ u 


_l 


► 


High 
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SWAT  requires  a  rating  of  1,  2,  or  3,  corresponding  to  low,  medium,  or  high  on  each  of 
the  dimensions  (time,  mental  effort,  and  stress)  for  each  measurement  event.  Using 
conjoint  measurement  techniques,  SWAT  ratings  can  be  converted  to  the  0-to-100 
interval  score.  SWAT  requires  a  sort,  from  low  to  high,  of  the  27  different  cards 
containing  all  the  possible  combinations  of  time,  mental  effort,  and  stress  dimensions. 
The  card  sort  is  completed  before  the  event  scoring  begins.  The  card  sort  provides  the 
necessary  information  for  the  conjoint  scaling  solution  that  allows  assigning  the  0-to-100 
score  for  the  event  ratings,  1-1-1  through  3-3-3.  A  given  event  rating,  such  as  2-1-3 
(medium  rating  on  time,  low  rating  on  mental  effort,  and  high  rating  on  stress),  can  then 
be  converted  into  a  score  that  has  a  value  ranging  from  0-to-100. 

The  NASA-TLX  uses  six,  20-point  low  to  high,  bipolar  scales  for  mental  demand,  physical 
demand,  temporal  demand,  performance,  effort,  and  frustration.  The  customizing 
portion,  weighting,  of  the  NASA-TLX  was  applied  after  the  event  scoring.  The  weighting 
is  done  to  establish  a  priority  among  the  dimensions  in  a  quantitative  fashion.  The 
weights  are  ranked  in  order  of  importance  by  a  forced-choice  paired  comparison  task. 
The  weights  are  combined  with  the  event  ratings  to  form  the  0-to-100  combined 
workload  score. 

All  of  the  subjective  ratings  were  collected  post-flight,  utilizing  video  tape.  Each 
simulation  flight  was  video  taped.  The  video  tape  recorded  a  quad  image  which 
contained:  (1)  right  side  profile  of  Captain,  upper  left  of  quad  image,  (2)  left  side  profile  of 
First  Office,  upper  right  of  quad  image,  (3)  left  three-quarter  view  of  Flight  Engineer 
panel,  lower  left  of  quad  image,  and  (4)  view  forward  of  flight  deck  from  pedestal,  lower 
right  of  quad  image.  Subjects  viewed  the  video  tape  at  the  end  of  a  day's  session  for  the 
purpose  of  making  subjective  ratings.  To  demarcate  the  measurement  windows  small 
light  emitting  diode  (LED)  lights  visible  in  the  video  tape  were  illuminated  during  the 
measurement  windows.  The  LED  lights  are  out  of  the  pilot's  field  of  view  during  the 
actual  simulation  runs.  For  the  purpose  of  making  subjective  ratings  the  pilots  were 
asked  to  attend  to  their  workload  when  the  LED  lights  where  illuminated.  When  the 
measurement  window  closed,  the  lights  were  extinguished,  the  video  tape  was  stopped 
by  the  experimenter,  and  the  subject  was  asked  to  make  event  ratings.  The  pilots  were 
instructed  not  to  refer  to  previous  ratings  when  making  event  ratings. 

PHYSIOLOGICAL  INSTRUMENTATION 

Data  was  collected  for  horizontal  and  vertical  eye  movement,  eyeblink  rate,  heart  rate, 
and  heart  rate  variability. 

The  pilots  had  electrocardiogram  (ECG)  electrode  leads  applied  to  the  chest  to  record 
heart  rate.  Conventional  stress-type  hospital  grade  disposable  silver-silver  chloride 
electrodes  were  used.  Since  only  the  peak  of  the  R-wave  was  relevant,  not  the  complete 
ECG  wave  form  morphology,  placement  was  mainly  dictated  by  considerations  of 
convenience.  One  lead  was  placed  just  above  the  sternum  and  another  was  placed 
approximately  four  centimeters  above  the  waist  and  ten  centimeters  to  the  left  of  the 
sagital  plane. 

Electrodes  to  record  the  electro-occulogram  (EOG),  including  eyeblinks,  were  placed  in 
a  conventional  manner:  active  and  referent  just  beyond  the  outer  canthus  of  the  left  and 
right  eyes  to  record  horizontal  movements,  and  active  and  referent  above  and  below  the 
left  eye  to  record  vertical  eye  movements  and  eye  blinks.  Pilots  were  instrumented  with 
Beckman  11mm  Silver-Silver  Chloride  mini-cup  electrodes,  held  on  by  adhesive  collars. 
Methyl  cellulose  was  used  as  the  electrode  cream.  Linked  mastoids  leads  served  as  the 
ground. 
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Resistance  readings  were  checked  for  all  the  leads.  Electrode  leads  were  allowed  to 
have  a  maximum  resistance  of  30  Kohms. 

All  the  leads  to  the  electrodes  were  connected  to  a  Grass  Instruments  isolator  electrode 
board  (Model  IMEB2).  The  connector  box  was  suspended  on  the  pilot’s  chest  by  cords 
which  were  tied  around  his  neck  and  waste.  The  subjects  reported  that  the  isolator  box 
did  not  interfere  with  their  flight  deck  activities. 

A  25  foot  long  Grass  Instruments  cable  led  from  this  box  to  the  array  of  Grass  amplifiers 
and  power  supply,  which  were  in  a  19  inch  rack  cabinet  fastened  to  the  back  wall  of  the 
simulator  cabin.  The  heart  signal  was  amplified  by  a  Grass  model  7P51 1  AC  amplifier 
and  the  two  EOG  signals  were  amplified  by  two  model  7P122  DC  amplifiers.  These  three 
physiological  signals,  along  with  an  event  marker  signal  and  the  audio  from  a 
microphone  on  the  subject's  label,  were  all  recorded  on  a  Hewlett  Packard  model  3968-A 
8  channel  FM  tape  recorder.  The  tape  speed  was  3  3/4"  per  second. 

* 

The  tape  recorder  was  under  computer  control.  The  computer  would  activate  the  FM 
tape  recorder  when  a  measurement  window  opened,  and  then  stopped  the  recording 
when  the  measurement  window  closed. 

The  data  collected  during  the  simulations  was  subsequently  played  back  in  the  ' 

laboratory  for  reduction  and  analysis. 

EYEBLINKS 

The  analog  signals  for  both  vertical  and  horizontal  eye  movement  were  printed  out  on 
strip  charts,  from  which  raters  "scored"  the  vertical  record  to  determine  the  number  of 
eyeblinks  (Figure  7.1. 2.2-5).  Two  scorers  were  used,  and  an  objective  scoring  criteria 
developed,  to  insure  a  between  scorer  agreement  of  at  least  95%.  Determining  the  time 
elapsed  between  the  opening  and  closing  of  each  window  made  it  possible  to  compute 
eyeblink  rate  in  blinks  per  minute. 

HEART  RATE 

Interbeat  interval  is  a  measure  of  the  time  elapsed  between  heart  beats,  whereas  heart 

rate  refers  to  how  many  beats  occur  in  a  minute.  The  analog  heart  signal  waveform  was 

examined  for  inter-beat  interval  scoring.  A  voltage  sensitive  Schmitt  trigger,  connected 

to  a  MINC  (Digital  Electronic  Corp.)  minicomputer,  was  used  to  digitize  the  analog 

record  by  timing  and  recording  the  duration  of  the  intervals  between  successive  "R- 

waves"  of  the  cardiac  signal  in  milliseconds.  For  each  window  all  the  heart  inter-beat  » 

intervals  (IBI's),  in  msecs  (milliseconds),  were  saved  as  computer  files  for  later 

processing  and  analysis. 

Heart  rate,  in  beats  per  minute,  can  be  computed  using  the  following  formula:  (1000  /  R- 
R  Interbeat  interval)  *  60.  The  transformation  of  the  IBI  to  beats  per  minute  is  non-linear.  > 

The  ordinal  position  of  the  phases  of  flight  ranked  by  workload  would  remain  the  same, 
but  the  transformed  scores  would  not  correlate  perfectly. 

Occasionally  heart  data  was  contaminated  by  movement  artifacts.  It  would  result  in  IBI 
values  significantly  different  with  the  pattern  of  IBI  values  collected  in  the  same 
measurement  window.  A  simple  filter  in  the  software  excluded  IBI  values  33%  different 
than  the  low  and  high  IBI  values  considered  to  be  valid  within  a  measurement  window. 
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HEART  RATE  VARIABILITY 

A  variety  of  methods  have  been  suggested  to  index  heart  rate  variability.  However,  as 
there  is  no  consensus  agreement  in  the  field  upon  any  one  best  way,  and  since  the 
simple  heart  rate  standard  deviation  has  often  been  used  for  this  purpose,  we  elected  to 
utilize  it  as  a  measure  of  the  heart  rate  variability. 

POWER  SPECTRAL  ANALYSIS 

A  power  spectral  analysis  of  the  heart  record  was  also  conducted  using  software 
provided  by  Randall  Harris  and  Allen  Pope  of  NASA-Langley.  Fourier  analysis  of  the  low- 
pass  filtered  cardiac  event  sequence  provided  a  spectrum  of  frequency  components  in 
the  range  of  0  Hz  to  0.5  Hz.  Two  bands  of  energy  from  the  spectrum  were  examined, 
the  bands  were  0.05  Hz  to  0.15  Hz  and  0.20  Hz  to  0.40  Hz.  The  area  between  0.05  and 
0.15  is  believed  to  reflect  changes  in  blood  pressure,  and  is  predicted  to  decrease  when 
the  pilot  is  engaged  in  a  cognitive  task  (Mulder,  1977).  The  area  between  0.20  Hz  and 
0.40  Hz  is  believed  to  reflect  changes  in  respiration  (Mulder,  1977). 

PERFORMANCE  DATA 

Data  from  the  simulator  was  collected  during  the  measurement  windows  as  well.  Wheel, 
column,  pedal,  and  throttle  position  data  was  collected  in  order  to  compute  control  input 
activity.  In  addition,  altitude  over  the  outer  and  middle  markers,  flight  director  deviation, 
glideslope  and  localizer  deviation,  and  lateral  deviation  from  runway  centerline  were 
collected  in  the  Approach  and  Landing  windows.  All  performance  data  was  collected  at 
a  rate  of  10  Hertz  (Figure  7.1. 2.2-6). 

Root  Mean  Square  error  of  flight  director  deviation,  as  well  as  localizer  and  glide  slope 
deviation  could  not  discriminate  between  low  and  high  workload,  nor  did  the  measures 
demonstrate  any  evidence  of  reliability.  The  piloting  task  in  commercial  aviation  has 
large  tolerances  in  the  precision  required  in  the  flying  task  (e.g.,  plus  or  minus  300  feet  at 
altitude).  Using  RMS  tracking  error  measures  as  indices  of  workload  would  require  the 
pilots  to  fly  with  a  level  of  precision  not  normally  required  in  revenue  service. 

CONTROL  INPUT  ACTIVITY 

Position  information  of  the  flight  controls:  Wheel,  column,  and  pedals  was  transformed 
into  a  measure  of  control  activity  labelled  control  input  activity.  The  algorithm  for 
reducing  the  control  position  information  to  control  inputs  uses  a  "counter"  which 
increments  if  the  difference  between  consecutive  control  positions  is  greater  than  2.5% 
of  the  total  range  of  travel  for  that  particular  control.  Since  the  measurement  windows 
cover  different  lengths  of  time  the  control  activity  is  divided  by  units  of  time  to  yield 
control  input  activity  per  minute. 

There  was  a  problem  in  losing  control  input  data  in  the  low  workload  condition.  If  the 
pilot  placed  the  plane  on  autopilot,  the  data  for  control  position,  and  consequently 
control  input,  was  not  collected.  There  are  measurement  windows  in  the  low  workload 
condition  which  do  not  contain  much  data,  making  the  comparison  of  low  and  high 
workload  conditions  meaningless  from  a  statistical  point  of  view.  Yet,  with  the  missing 
data  there  is  still  a  strong  indication  that  control  input  activity  can  discriminate  low  and 
high  workload  conditions. 
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CONTROL  INPUT  ACTIVITY 


Figure  7. 1.2. 2-6 
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A  secondary  task  was  also  implemented  to  measure  pilot  spare  capacity  (spare  capacity 
is  thought  to  be  correlated  with  workload). 


Two  measures  were  derived  from  the  Secondary  Task:  response  time  to  the  positive 
probe  and  probe  accuracy  (percentage  of  positive  probes  responded  to  correctly).  A 
constraint  was  imposed  on  the  time  available  for  subjects  to  respond.  Responses  which 
took  longer  than  10  seconds  were  disregarded  because  the  task  was  not  executed 
immediately. 

SECONDARY  TASK  RESPONSE  TIME 

ATC  would  start  a  timer  at  the  beginning  of  their  utterance  when  they  issued  calls  to 
aircraft  designated  as  positive  probes.  The  positive  probes  were  the  pilot's  own  aircraft 
(For  example,  “American  247  contact  departure  control  ..."),  and  another  company 
aircraft  in  the  area  (For  example,  "American  241  descend  and  maintain  ...").  Pilots  were 
instructed  to  press  the  push-to-talk  (PTT)  switch  in  the  left  side  of  the  yoke  when  they 
heard  a  positive  probe  aircraft  called  by  ATC.  The  PTT  switch  stopped  the  timer  ATC 
had  activated.  The  pilot  was  allowed  to  write  down  the  call  signs  of  both  positive  probe 
aircraft,  and  had  the  call  signs  available  to  them  throughout  the  flight.  All  other  ATC 
initiated  calls  to  aircraft  were  designated  as  negative  probes. 

There  were  two  sets  of  call  signs  for  the  positive  probes  (241  and  247  &  352  and  356). 
The  call  signs  were  counter-balanced  across  workload  condition  (low  and  high)  and 
route  (San  Francisco-Stockton  and  Sacramento-San  Francisco). 

SECONDARY  TASK  PROBE  ACCURACY 

Probe  accuracy  is  the  percentage  of  probes  correctly  responded  to  per  measurement 
window.  Again,  the  flaws  in  the  positive  probing  by  ATC  allows  only  for  a  description  of 
trends  in  the  data. 

7.1.3  PROCEDURE  AND  EQUIPMENT 

SIMULATOR 

The  Man-Vehicle  System  Research  Facility  (MVSRF)  B-727  six  degree  of  freedom  motion 
base  simulator  was  used  to  provide  Phase  II  certified  fidelity.  The  MVSRF  has  a  number 
of  assets,  most  notably  is  the  ability  to  simulate  the  Air  Traffic  Control  (ATC) 
environment.  The  ATC  simulation  creates  realistic  levels  of  communications  workload,  a 
part  of  the  standard  commercial  transport  environment  often  overlooked  in  simulation. 

SCENARIO 

There  were  two  routes  flown  each  session.  The  routes  were  Sacramento  -  to  -  San 
Francisco  (SMF-SFO)  and  San  Francisco  -  to  -  Stockton  (SFO-SCK).  Both  flights  were 
flown  at  1 1 ,000  feet  enroute,  to  ILS  approaches  and  landings. 

COUNTER-BALANCE  OF  CONDITIONS 

The  presentation  of  routes  is  partially  counter-balanced  across  pilots  with  low  and  high 
workload  flights.  For  half  of  the  subjects  the  SMF-SFO  and  SFO-SCK  are  the  low  and 
high  workload  flights,  respectively.  For  the  other  half  of  the  subjects  the  combination  of 


flights  and  workload  is  reversed,  SMF-SFO  and  SFO-SCK  are  the  high  and  low  workload 
flights,  respectively. 

Pilots  flew  two  scenarios,  a  high  and  a  low  workload  flight,  on  both  visits.  Pilots  were 
randomly  assigned  to  either:  1)  SFO-SCK  (high  workload)  and  SMF-SFO  (low  workload), 
or  2)  SFO-SCK  (low  workload)  and  SMF-SFO  (high  workload).  Half  the  subjects  flew  the 
high  workload  scenario  first,  the  other  half  of  the  subjects  flew  the  low  workload 
condition  first.  On  the  retest  day  pilots  flew  the  exact  same  scenario,  departure-arrival 
destinations  and  workload  condition,  as  the  previous  session  but  the  scenario  order  was 
reversed. 

DAILY  SCHEDULE 

Subject  pilots  arrived  at  the  MVSRF  facility  at  9:00  am.  Subjects  were  greeted  by  the 
experimenter  and  tape  recorded  instructions  regarding  the  day's  activities  were  played 
for  them.  Subjects  then  received  “differences  training"  from  Preston  Suit  (Boeing  Flight 
Crew  Training).  The  differences  training  involved  a  discussion  of  the  configuration  of  the 
cockpit.  The  differences  training  was  facilitated  by  using  full-size  color  photographs  of 
the  instrument  panels  in  the  simulator.  After  differences  training  was  completed  the  pilot 
was  briefed  on  the  routes  and  weight  &  balance  of  the  aircraft  for  the  simulation  runs. 

Following  the  differences  training,  tape  recorded  instructions  for  the  SWAT  card  sort  was 
played  for  the  pilot.  The  experimenter  provided  further  clarification  on  the  technique 
used  for  the  SWAT  card  sort.  The  card  sort  required  20  to  60  minutes  to  complete. 

The  pilot  and  experimenter  then  went  to  lunch. 

Following  lunch  the  pilot  was  instrumented  with  the  physiological  equipment  and  placed 
in  the  simulator  cab.  The  pilot  then  departed  San  Francisco  International  Airport  on 
runway  28R  and  spent  time  flying  the  aircraft  "around  the  pattern."  The  pilot  was 
encouraged  to  practice  steep  turns,  pull  the  throttle  back  on  an  engine,  and  generally 
get  comfortable  with  the  handling  qualities  of  the  simulator.  The  pilot  flew  an  ILS 
approach  to  a  touch  and  go  on  28R  at  SFO.  The  pilot  again  flew  "around  the  pattern"  to 
another  ILS  approach  and"  landing  on  28R  at  SFO. 

The  pilot  then  flew  the  two  test  trials.  Following  the  test  runs  the  instrumentation  was 
removed  from  the  pilot.  Tape  recorded  instructions  on  event  rating  for  either  SWAT  or 
the  NASA-TLX  was  then  played  for  the  pilot.  Then  the  videotape  of  the  simulation  test 
runs  was  played  for  the  pilot  in  order  to  obtain  subjective  event  ratings.  Only  a  small 
segment  (30  seconds)  prior  to  the  actual  measurement  window,  and  the  window  itself 
was  played  for  the  purpose  of  making  the  event  ratings. 

Finally,  the  subject  pilot  was  thanked  for  participating. 

7.1 .4  STATISTICAL  ANALYSES 

Even  though  there  are  numerous  statistical  comparisons  made,  each  workload 
assessment  technique  is  treated  as  though  it  was  the  only  dependent  variable  utilized  in 
the  study.  No  adjustment  was  made  to  the  alpha  level  (probability  of  a  Type  I  error)  for 
the  various  comparisons  reported  from  the  simulation  effort.  This  approach,  referred  to 
by  Kirk  (1982)  as  a  "contrast-wise  Type  I  error  rate"  may  seem  to  be  libera!.  The  reason 
for  the  contrast-wise  Type  I  error  rate  is  that  the  statistical  effects  tested  were  predicted  a 
priori,  and  a  stringent  Type  I  error  rate  is  applied  throughout.  A  nominal  alpha  level  of 
0.01  was  adopted  for  determining  significance  for  all  the  analyses.  Alpha  levels  between 
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0.01  and  0.05  were  considered  as  strong  trends  while  alpha  levels  between  0.05  and 
0.10  were  considered  trends. 

Boeing's  TLA  was  utilized  to  confirm  that  the  manipulations  of  the  pilot's  task  demands 
would,  in  fact,  yield  a  change  in  the  pilot's  workload  between  the  “low"  and  "high" 
workload  flights. 

It  is  necessary  to  establish  a  decision  rule  to  be  able  to  determine  if  a  given  workload 
measure  has  shown  validity  and  reliability.  The  threshold  of  the  decision  rule  will  most 
likely  generate  an  argument  as  to  the  appropriateness  of  the  threshold,  but  the  rule  is 
necessary  for  a  discussion  of  the  "goodness"  (validity  and  reliability)  of  the  measure. 

7.1. 4.1  VALIDITY  ANALYSES 

In  order  claim  validity  for  any  of  the  assessment  techniques  it  was  necessary  for  the 

workload  measure  to  discriminate  between  the  low  and  high  workload  conditions.  A  2  X  * 

2X7  repeated  measures  ANOVA  was  performed  for  each  workload  measure.  The 

factors  of  the  2  X  2  X  7  ANOVA  were  session  (1  or  2),  workload  level  (low  or  high),  and 

phase  of  flight  (seven  measurement  windows),  respectively.  The  means  represented  in 

the  graphs  are  from  the  2  X  2  X  7  ANOVAs.  % 

An  unfortunate  artifact  of  the  repeated  measures  ANOVA  approach  is  the  case-wise 
deletion  of  subjects  owing  to  missing  data  in  any  of  the  measurement  windows.  If  a 
subject  fails  to  have  data  in  any  of  the  28  measurement  windows  (2  sessions  containing 
2  flights  with  7  measurement  windows  per  flight)  for  a  given  workload  measure  then  the 
entire  subject  is  deleted  from  the  analysis. 

All  significant  F  ratios  are  reported,  and  the  results  of  the  test  for  a  main  effect  of 
workload  will  be  reported  whether  there  is  a  statistically  significant  finding  or  not. 

An  a  priori  prediction  is  that  workload  would  vary  across  the  phases  of  flight,  for  either 
the  low  or  high  workload  flight.  Although  a  workload  measure  may  not  be  able  to 
discriminate  among  the  periods  of  high  workload  in  a  flight  (i.e.,  takeoff,  approach,  and 
landing),  a  workload  measure  should  be  able  to  discriminate  low  from  high  within  a  flight 
(cruise  and  landing).  Oneway  ANOVAs  were  computed,  for  both  the  low  and  high 
workload  flights,  for  each  workload  assessment  technique  to  determine  if  the  various 
phases  of  flight  could  be  discriminated  from  one  another. 

A  simple  approach  will  be  taken  in  examining  the  discriminability  of  phase  of  flight  by  a 
workload  measure.  Utilizing  a  paired-comparison  approach  the  various  phases  of  flight  » 

were  compared  to  one  another  using  the  Newman- Kuels  range  statistic. 

7.1. 4.2  VALIDITY  DECISION  CRITERIA 

In  order  for  a  workload  measure  to  demonstrate  validity  the  measure  should  find  a  main  f 

effect  (discriminate)  for  the  workload  factor.  Since  there  is  an  a  priori  prediction  for  a 
difference  in  workload  for  the  phases  of  flight,  and  the  malfunctions  in  the  high  workload 
condition  will  effect  the  low  workload  windows  in  that  flight,  an  interaction  of  workload 
and  phase  of  flight  should  be  present  as  well.  When  the  malfunctions  occur  the 
workload  should  be  significantly  higher  than  the  corresponding  windows  in  the  low 
workload  condition.  A  significant  workload  by  window  interaction  demonstrates  that  not 
all  windows  are  significantly  different  between  the  low  and  high  workload  conditions.  A 
main  effect  of  workload  along  with  a  workload  by  phase  of  flight  interaction  indicates 
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appropriate  sensitivity  of  the  assessment  technique  to  the  manipulation  of  levels  of 
workload. 

An  interaction  of  test  session  (day  1  or  2)  with  another  factor  (e.g.,  workload  or  phase  of 
flight)  could  indicate  a  number  of  influences  including: 

(a)  instability  of  the  measure, 

(b)  practice  effect, 

(c)  adaptation  to  test  conditions. 

No  systematic  attempt  was  made  to  describe  the  nature  of  the  Newman-Kuels  range 
statistic  due  to  complications  arising  from  the  large  number  of  comparisons.  Our 
approach,  simply  stated,  is  the  more  significant  differences  that  were  found  the  more 
discriminable  the  workload  measure  was  thought  to  be.  (A  note  to  the  reader:  Even  the 
results  for  the  various  body  channels  of  the  TLA  disassociate,  leaving  no  clear  answer  in 
terms  of  a  prediction  for  global  workload  differences  between  the  different  phases  of 
flight.  Therefore  a  detailed  interpretation  of  phase  of  flight  differences  for  the  various 
workload  measures  is  beyond  the  scope  of  this  project.) 

7.1. 4.3  RELIABILITY  ANALYSES 

A  test/retest  methodology  was  employed  so  that  the  reliability  of  the  various  measures 
could  be  assessed. 

In  order  to  compute  the  correlation  coefficients,  each  measurement  window  was 
examined  individually.  The  result  of  the  test/retest  evaluation  yields  14  different 
correlation  coefficients  for  the  7  different  measurement  windows  for  the  2  different,  low 
and  high  workload,  flights.  For  each  workload  measure,  the  correlations  were  computed 
by  pairing  the  session  1  and  2  scores  for  all  the  pilots.  This  approach  allows  us  to 
examine  the  reliability  of  a  workload  measure  under  the  conditions  for  a  specific  phase  of 
flight. 

An  alternative  method  for  evaluating  test/retest  reliability  would  be  to  examine  the 
session  1  to  session  2  pairings  of  workload  scores  for  the  14  different  windows 
separately  for  each  subject.  This  method  was  not  chosen  because  the  result  only 
describes  the  reliability  of  a  workload  measure  for  different  subjects.  It  is  already  known 
that  some  subjects  are  capable  of  more  reliable  assessments  than  are  other  subjects. 
The  method  employed  in  the  current  study,  for  computing  the  test/retest  correlation 
coefficient,  allows  for  the  examination  of  the  reliability  of  a  measure  for  certain  test 
conditions  (task  demands  inherent  in  a  given  phase  of  flight)  across  a  variety  of  subjects. 

Unlike  the  artifact  of  case-wise  deletion  found  for  the  repeated  measures  ANOVAs,  the 
correlation  coefficients  were  computed  on  as  many  complete  pairs  of  scores  as  were 
available.  The  principle  justification  for  the  unequal  samples  contributing  to  the  various 
correlation  coefficients  is  the  idea  of  having  as  many  scores  as  possible  contribute  to 
each  correlation.  The  idea  of  central  tendency  dictates  that  all  the  scores  available  from 
the  sample  should  be  used,  when  possible,  to  provide  a  more  reliable,  or  stable, 
estimate  of  the  population. 

Additionally,  for  each  workload  measure  an  estimation  of  inter-rater  reliability  was  derived 
by  correlating  each  pilot's  score  for  the  14  measurement  windows  (average  of  session  1 
and  2)  with  the  group  average  for  the  corresponding  windows.  Inter-rater  reliability  is 
then  expressed  as  a  percentage  of  the  pilots  that  show  a  significant  correlation  with  the 
group  means  for  the  14  measurement  windows. 
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7.1 .4.4  RELIABILITY  DECISION  CRITERIA 


To  demonstrate  reliability  a  workload  measure  should  find  significant  test/retest 
correlations  for  the  various  phases  of  flight.  Unfortunately  there  is  variation  in  the  sample 
size  for  the  different  workload  measures.  There  are  sample  sizes  of  9, 16,  and  17  for  the 
subjective,  physiological,  and  performance  measures,  respectively.  The  different  sample 
sizes  influences  the  critical  Pearson  coefficient  necessary  for  a  significant  correlation.  It 
is  therefore  very  difficult  to  establish  a  uniform  decision  criteria  regarding  the  “goodness" 
of  workload  measures'  reliability.  A  good  rule  of  thumb  is  that  each  workload  measure 
demonstrate  positive  correlations  that  are  large  (nearer  to  + 1 .0  than  0). 

For  confidence  about  the  inter-rater  reliability  criteria  was  established  that  at  least  half  of 
the  pilots  scores  (average  test/retest)  for  the  14  measurement  windows  should  be 
significantly  correlated  with  the  group  means. 

7.2  RESULTS 

7.2.1  TIMELINE  ANALYSIS  RESULTS 

To  compute  the  percentage  of  time  required,  the  total  time  used  by  a  particular  channel 
(visual,  manual,  auditory,  verbal  or  cognitive)  for  the  measurement  period  was  divided  by 
time  available. 

There  was  no  change  in  procedures  during  the  takeoff  window  between  the  high  and  low 
workload  segments  in  the  scenarios  built  for  TLA.  Therefore,  for  all  channels,  visual, 
manual,  verbal,  auditory  and  cognitive,  the  data  shown  for  the  takeoff  window  is  identical 
for  high  and  low  workload. 

The  data  on  the  visual  channel  (Table  7.2.1-1)  shows  that  visual  demands  are  higher  in 
the  high  workload  scenarios  starting  at  the  climb  window,  when  the  pilot  attempts  to 
engage  the  autopilot.  The  higher  demand  on  the  visual  channel  continues  through  the 
top  of  climb  when  the  engine  failure  occurred,  and  cruise  when  the  hydraulic  failure 
occurred.  The  visual  demands  remain  higher  through  descent,  approach,  and  landing 
as  the  pilot  is  manually  flying  the  aircraft  in  low  visibility  for  a  15  degree  flap  landing. 

Manual  data  is  computed  in  TLA  for  the  left  hand,  right  hand,  and  total  manual  data  (sum 
of  the  two  hands).  The  assumption  was  made  in  preparing  this  data  that  there  are 
continuous  corrections  (about  once  per  second)  made  with  the  left  hand  in  the  high 
workload  scenarios,  except  in  the  cruise  phases  when  the  aircraft  is  trimmed,  then  the 
correction  rate  becomes  once  every  5  seconds.  These  corrections  do  not  occur  in  the 
low  workload  scenarios  when  the  autopilot  is  engaged.  The  differences  in  workload 
between  high  and  low  scenarios  in  the  manual  channel  (Table  7.2. 1-1)  occurs  starting  at 
the  top  of  climb  with  the  engine  failure.  In  the  high  workload  scenarios  the  pilot  must 
take  actions:  the  engine  failure/shutdown  checklists  are  read,  rudder  is  trimmed,  and 
thrust  is  adjusted.  In  cruise,  in  the  low  workload  scenario,  when  the  autopilot  is  engaged 
there  is  a  very  low  manual  requirement.  In  the  high  workload  scenario  in  cruise, 
however,  the  right  hand  must  take  actions  as  a  result  of  the  "B"  system  hydraulics  failure. 
Descent,  approach  and  landing  show  a  much  higher  manual  requirement  in  the  high 
workload  scenario  as  might  be  expected.  The  pilot  is  manually  flying  the  aircraft  and 
must  respond  to  checklists  required  with  the  failure  conditions.  The  manual  sum 
requirement  is  over  100%  in  approach  and  landing.  The  pilot  is  using  both  right  and  left 
hands  at  the  same  time  and  manual  sum  reflects  this. 

The  verbal  component  (Table  7.2. 1-1)  in  these  flights  was  low  for  the  pilot.  All 
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Table  7.2. 1-1 


WINDOW 

VISUAL 

MANUAL 

LEFT 

MANUAL 

RIGHT 

VERBAL 

AUDITORY 

COGNITIVE 

TAKEOFF 

88 

65.5 

20 

5 

16 

73 

CLIMB 

97 

58.5 

28.5 

10 

51.5 

100 

TOC 

61.5 

0 

14 

2 

11 

38.5 

CRUISE 

30 

0 

1 

0 

1.5 

32 

TOD 

60 

2.5 

14 

12.5 

34 

57.5 

APPROACH 

65.5 

3 

15.5 

5 

19 

36.5 

LANDING 

86 

61 

38 

2 

14.5 

41.5 

HIGH 

WORKLOAD 

WINDOW 

VISUAL 

MANUAL 

LEFT 

MANUAL 

RIGHT 

VERBAL 

AUDITORY 

COGNITIVE 

TAKEOFF 

88 

65.5 

20 

5 

16 

73 

CLIMB 

100 

58.5 

26.5 

13 

54.5 

100 

TOr 

78.5 

13 

70.5 

14.5 

38.5 

97.5 

CRUISE 

37 

1 1 

2 

10.5 

13 

37.5 

TOD 

62 

59.5 

18 

12.5 

28.5 

60.5 

APPROACH 

72 

57.5 

16.5 

5 

17.5 

47.5 

LANDING 

100 

61 

56.5 

2.5 

18.5 

50 
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communications  with  ATC  and  dispatch  were  handled  by  the  first  officer.  Pilot  verbal 
communications  were  limited  to  crew  coordination  type  communications.  A  small 
difference  occurred  in  climb  because  the  autopilot  did  not  engage  in  the  high  workload 
scenarios.  At  the  top  of  climb  the  pilot  makes  the  decision  to  shut  down  the  engine  and 
asks  the  first  officer  to  advise  ATC  of  the  failure.  In  cruise  the  increase  in  verbal 
workload  occurs  when  the  hydraulics  failure  checklist  is  called  for.  Though  the  percent 
of  communication  in  the  high  and  low  workload  scenarios  in  descent  and  approach  is 
the  same,  the  content  is  not.  In  the  high  workload  scenario  the  pilot  is  handing  off  more 
tasks  as  he  manually  flies  the  aircraft. 

For  the  auditory  channel  the  significant  changes  between  the  low  and  high  workload 
flights  occur  at  the  top  of  climb  and  in  the  cruise  windows  when  the  checklists  are  called 
for  after  the  malfunctions  occur.  In  descent  and  approach  we  see  a  slight  reversal  due 
to  the  pilot  taking  more  of  a  command  role,  as  discussed  in  the  verbal  data. 

The  cognitive  channel  is  influenced  by  the  number  of  indicators  looked  at  by  the  pilot,  * 

and  the  complexity  both  of  the  indicators  and  the  procedures  performed.  In  other  words, 
how  long  it  takes  the  pilot  to  act  is  based  on  the  number  of  choices  available.  In  takeoff 
and  climb  the  cognitive  workload  is  the  same  for  the  high  and  low  scenarios  (T able  7.2.1- 
1).  In  climb  the  cognitive  channel  is  at  100%  due  to  reconfiguration  tasks,  thrust 
management,  heading  changes,  and  completion  of  the  required  check  lists.  In  addition,  * 

in  the  high  workload  scenario,  the  autopilot  fails  to  engage.  At  the  top  of  climb  (high 
workload)  the  cognitive  channel  is  high  because  of  the  engine  failure.  In  the  high 
workload  flight  during  cruise  the  hydraulics  failure  occurs,  and  the  pilot  calls  for  the 
appropriate  checklist.  In  descent,  approach,  and  landing  there  is  a  higher  cognitive  load 
in  the  high  workload  scenarios  as  a  result  of  manually  flying  the  aircraft  with  an  increase 
in  system  monitoring  and  more  frequent  instrument  scanning. 

The  two  low  workload  scenarios  were  averaged  tq  yield  the  low  workload  scores  for  the 
various  body  channels.  Similarly,  the  two  high  workload  scenarios  were  averaged  to 
yield  the  high  workload  scores  for  the  various  body  channels  (Table  7.2. 1-1).  Using 
percentages  negates  the  fact  that  each  of  the  measurement  periods  in  the  simulation 
was  of  varying  length  (ranging  from  one  minute  to  over  6  minutes). 

7.2.2  SUMMARY  OF  RESULTS 

The  task  analysis  provides  a  link  between  the  past  certification  workload  assessment 

methods  and  the  workload  assessment  methods  tested  in  the  Part-Task  simulation.  The 

comparison  of  workload  assessment  methods  provides  the  opportunity  to  assess  the 

validity  of  the  "pilot  in  the  loop"  methods  against  an  analytic  tool,  namely  TLA.  If  a  new  * 

workload  measure  agrees  with  the  task  analyses,  both  probably  reflect  the  same 

conditions  (same  task-demands).  If  the  timeline  analysis  and  the  workload  measure  do 

not  agree,  but  the  measure  is  shown  to  demonstrate  validity  and  reliability,  then  the 

measure  may  reflect  a  type  of  workload  not  accurately  quantified  by  older  techniques. 

SWAT,  NASA-TLX,  the  1-to-20  point  Overall  Workload  Score,  and  Heart  Rate  all 
demonstrated  evidence  of  validity  by  discriminating  workload,  and  a  significant 
interaction  of  workload  and  phase  of  flight. 

NASA-TLX,  the  1-to-20  point  Overall  Workload  Score,  Eyeblink,  and  Heart  Rate  all 
demonstrated  evidence  of  reliability  by  finding  significant  test/retest  correlations.  In 
addition,  at  least  50%  of  the  pilots  scores  correlated  significantly  with  the  group  mean  for 
the  workload  measure. 
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7.2.2. 1  SUBJECTIVE  RATINGS 


SWAT,  NASA-TLX,  and  the  1-to-20  point  overall  workload  score  demonstrated  evidence 
for  validity  by  discriminating  between  low  and  high  workload.  In  addition,  the  NASA-TLX 
and  the  1  -to-20  point  Overall  Workload  Score  demonstrated  evidence  of  being  reliable 
measures. 

SWAT 

To  yield  the  appropriate  0  to  100  scaling  solution  for  the  SWAT  ratings,  the  group  scaling 
solution  was  used  for  the  SWAT  card  sort.  The  Kendall’s  coefficient  of  concordance 
comparing  the  ranks  of  the  card  sorts  for  the  various  pilots  was  greater  (0.7824)  than  the 
recommended  0.78  for  using  the  group  scaling  solution. 

SWAT  discriminated  between  the  low  and  high  workload  flights,  F(1,8)  =  17.20, 
(MSe  =  1313,  pc.01)  (Figure  7.2.2.1-1  and  Table  7.2.2.1-1).  A  workload  by  phase  of 
flight  interaction  was  significant  as  well,  F(6, 48) =9.58,  (MSe  =  156,p<.01). 

A  strong  trend  for  a  main  effect  for  phase  of  flight  discrimination  was  found, 
F(6,48)  =  2.93,  (MSe=298,  p<.02).  No  main  effect  was  found  for  a  oneway  ANOVA 
examining  phase  of  flight  discrimination  for  the  low  workload  flight,  F(6,48)  =  1.55,  ns.  A 
significant  main  effect  was  found  for  a  oneway  ANOVA  examining  phase  of  flight 
discrimination  in  the  high  workload  condition,  F(6,48)=7.55,  (MSe  =  139,  p<.01).  A 
Newman-Kuels  range  statistic  was  computed  to  determine  SWATs  ability  to  discriminate 
phase  of  flight  for  the  high  workload  flight  5  out  of  21  comparisons  were  significant. 

In  evaluating  the  test/retest  reliability  of  SWAT  2  significant  correlations,  out  a  possible 
14,  were  found  (Table  7.2.2. 1-2). 

Examining  inter-rater  reliability  found  78%  of  the  subject's  scores  correlated  significantly 
with  the  means  for  the  1 4  measurement  windows. 

NASA-TLX 

To  yield  the  appropriate  0  to  100  workload  scaling  solution  for  the  NASA-TLX,  each 
individual's  weighting  scores  were  applied  to  their  event  ratings. 

NASA-TLX  discriminated  between  the  low  and  high  workload  flights,  F(1,7)  =  17.27, 
(MSe=436,p<.01)  (Figure  7. 2. 2.1-2  andTable  7.2. 2.1-3).  A  workload  by  phase  of  flight 
interaction  was  significant  as  well,  F(6, 42) =4.69,  (MSe  =  126,p<.01). 

A  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,42)=4.19,  (MSe=84, 
P<.01).  No  main  effect  was  found  for  a  oneway  ANOVA  examining  phase  of  flight 
discrimination  for  the  low  workload  flight,  F(6,42)  =  1 .75,  ns.  A  significant  main  effect  was 
found  for  a  oneway  ANOVA  examining  phase  of  flight  discr  imination  for  the  high 
workload  flight,  F(6,48)  =  7.14,  (MSe=68,  p<.01).  A  Newman-Kuels  range  statistic  was 
computed  to  determine  NASA-TLX's  ability  to  discriminate  phase  of  flight  conditions  for 
the  high  workload  flight  5  out  of  21  comparisons  were  significant. 

In  evaluating  the  test/retest  reliability  of  NASA-TLX  4  significant  correlations,  out  of  a 
possible  14,  were  found  (Table  7.2.2. 1-4). 

Examining  inter-rater  reliability  found  78%  of  the  subject's  scores  correlated  significantly 
with  the  means  for  the  14  measurement  windows. 
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Figure  7.2.2. 1-1 
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Table  7. 2. 2. 1-1 

Subjective  Workload  Assessment  Technique 

(SWAT) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

13.21 

(14.77) 

18.41 

(16.64) 

Climb 

17.52 

(17.53) 

24.59 

(19.04) 

Top  of  climb 

12.51 

(10.67) 

49.77 

(24.54) 

Cruise 

7.83 

(13.69) 

44.04 

(22.28) 

Top  of  descent 

14.67 

(9.47) 

29.74 

(18.04) 

Approach 

18.46 

(11.56) 

34.97 

(19.05) 

Landing 

18.53 

(10.84) 

33.72 

(20.94) 

Ml  IG7  15  H3dO 
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Table  7. 2. 2. 1-2 


SWAT 

Part  Task  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.46 

0.83* 

Climb 

0.44 

0.70 

Top  of  climb 

-0.01 

0.80* 

Cruise 

0.67 

0.46 

Top  of  descent 

-0.18 

0.41 

Approach 

0.10 

0.69 

Landing 

-0.0005 

0.48 

r(7)  =  .798* 
♦Significant  p<.01 
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Figure  7. 2. 2. 1-2 
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Table  7. 2. 2. 1-3 

NASA  -  Task  Load  Index 
(TLX) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 
Mean  SD 

High-workload  flight 
Mean  SD 

Takeoff 

16.50 

(7.75) 

19.28 

(8.69) 

Climb 

19.81 

(11.63) 

25.78 

(10.62) 

Top  of  climb 

17.88 

(9.87) 

39.78 

(16.78) 

Cruise 

12.19 

(4.14) 

38.50 

(20.57) 

Top  of  descent 

18.75 

(10.55) 

29.78 

(12.41) 

Approach 

16.94 

(9.05) 

28.39 

(10.96) 

Landing 

20.25 

(12.22) 

25.39 

(13.10) 
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Table  7. 2. 2. 1-4 


Window 

NASA-TLX 

Part  Task  Simulation 

Test-Retest 
Reliability  Correlations 

Low 

High 

Takeoff 

0.79 

0.61 

Climb 

0.41 

0.57 

Top  of  climb 

0.21 

0.56 

Cruise 

0.47 

0.82* 

Top  of  descent 

0.87* 

0.51 

Approach 

0.94* 

0.47 

Landing 

0.61 

0.83* 

r(7)  =  0.798* 
‘Significant  p  <  0.01 

f  1167.77  R7G 
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1  -to-2Q  POINT  OVERALL  WORKLOAD  SCORE 


The  1-to-20  point  workload  score  was  analyzed  without  any  sort  of  transformation  of  the 
event  ratings. 

The  1-to-20  overall  workload  score  discriminated  between  the  low  and  high  workload 
flights  f(1,8)=27.14,  (MSe  =  16,  p<.01)  (Figure  7.2.2.1-3  and  Table  7.2.2.1-5).  A 
workload  by  phase  of  flight  interaction  was  significant  as  well,  F(6,48)=7.41,  (MSe=5, 
P<01). 

A  main  effect  was  found  for  phase  of  flight  discrimination  was  found,  F(6,48)=4.15, 
(MSe=4.8,  p<  .01).  No  main  effect  was  found  for  a  oneway  ANOVA  examining  phase  of 
flight  discrimination  for  the  low  workload  flight,  F(6,48)  =  2.35,  ns.  A  significant  main 
effect  was  found  for  the  oneway  ANOVA  examining  phase  of  flight  discrimination  for  the 
high  workload  flight,  F(6,48)=8.22,  (MSe=3,  p<.01).  A  Newman-Kuels  range  statistic 
was  computed  to  determine  NASA-TLX's  ability  to  discriminate  phase  of  flight  for  the  high 
workload  flight  5  out  of  21  comparisons  were  significant. 

In  evaluating  the  test/retest  reliability  of  the  overall  workload  score  4  significant 
correlations,  out  of  a  possible  14,  were  found  (Table  7. 2. 2.1-6). 

Examining  inter-rater  reliability  found  78%  of  the  subject's  scores  correlated  significantly 
with  the  means  for  the  14  measurement  windows. 

7.2.2.2  PHYSIOLOGICAL  MEASURES 

Heart  rate,  as  measured  by  inter-beat  interval  and  the  respiration  component  of  the 
Power  Spectral  analysis  both  demonstrated  evidence  for  validity  by  discriminating 
between  low  and  high  workload.  In  addition,  eyeblink  rate  and  inter-beat  interval  both 
demonstrated  evidence  of  reliability. 

EYEBLINKS 

Eyeblink  rate  was  not  able  to  discriminate  low  and  high  workload  conditions,  F  <  1 
(Figure  7.2.2.2-1  and  Table  7.2.2.2-1).  A  session  by  workload  by  phase  of  flight 
interaction  was  significant,  F(6, 84)  =3.52,  (MSe=111,  p<.0l)  indicating  instability  of  the 
measure  over  time  as  well  as  some  sort  of  habituation  of  the  eyeblink  response. 

A  main  effect  for  phase  of  flight  discrimination  was  found,  F(6, 84)  =35.35,  (MSe=345, 
p<.01).  Two  separate  oneway  ANOVAs  were  perfc*med  and  significant  main  effects  for 
phase  of  flight  were  found  for  both  the  low  and  high  workload  flights,  F(6,96)  =  7.14, 
(MSe=20,  pc.01)  and  F(6,96)  =  10.61,  (MSe  =  23,  pc.01),  respectively.  Eyeblink  rate 
could  discriminate  3  out  of  21,  and  6  out  of  21,  phase  of  flight  comparisons  for  the  low 
and  high  workload  conditions,  respectively. 

The  eyeblink  data  demonstrated  good  test-retest  reliability,  8  out  of  a  possible  14 
correlations  were  significant  (Table  7. 2. 2. 2-2). 

Examining  inter-rater  reliability  found  56%  of  the  subjects  correlated  significantly  with  the 
means  per  measurement  window). 
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Figure  7. 2. 2. 1-3 
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Table  7. 2. 2. 1-5 

1-20  Point  Overall  Workload  Score 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 


Low-workload  flight 


High-workload  flight 


Mean 

SD 

Mean 

SD 

Takeoff 

3.22 

(2.06) 

3.94 

(2.19) 

Climb 

3.61 

(2.18) 

4.78 

(2.02) 

Top  of  climb 

3.39 

(2.10) 

8.50 

(3.22) 

Cruise 

2.22 

(0.97) 

7.94 

(4.28) 

Top  of  descent 

3.78 

(2.56) 

6.11 

(2.62) 

Approach 

3.39 

'  (2.13) 

6.00 

(2.82) 

Landing 

4.67 

(3.32) 

5.50 

(3.06) 

H1 167  17  H3od 
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Table  7. 2. 2. 1-6 


1-  to  20-Point  Overall  Workload 

Part  Task  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.77 

0.80 

Climb 

0.12 

0.37 

Top  of  climb 

0.28 

0.66 

Cruise 

0.45 

0.75 

Top  of  descent 

0.97* 

0.51 

Approach 

0.93* 

0.46 

Landing 

0.83* 

0.76 

r(7)  =  .798* 
‘Significant  p<.01 
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Figure  7.2.2. 2-1 

Eyeblink  Rate 
(Blinks  per  Minute) 

Part  Task  Simulation 
Mean  Scores 


Eyeblinks/ 
min  12 


Low  workload 
High  workload 


Takeoff  Climb  Top  of  Cruise  Top  of  Approach  Landing 
climb  descent 

(TOC)  (TOD) 

Phase  of  flight 


Table  7. 2. 2. 2-1 


Eyeblink  Rate  (Blinks  per  Minute) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

14.32 

(5.80) 

13.65 

(6.50) 

Climb 

10.18 

(5.97) 

9.26 

(6.05) 

Top  of  climb 

8.29 

(5.54) 

11.32 

(9.30) 

Cruise 

14.44 

(6.19) 

13.53 

(8.08) 

Top  of  descent 

10.53 

(6.60) 

9.00 

(4.85) 

Approach 

12.09 

.  (5.75) 

7.97 

(6.32) 

Landing 

16.41 

(6.20) 

18.94 

(8.43) 
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Table  7. 2. 2. 2-2 


Eyeblink  Rate  (Blinks  per  Minute) 

Part  Task  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.6 

0.7* 

Climb 

0.34 

0.63 

Top  of  climb 

0.84* 

0.84 

Cruise 

0.31 

0.75 

Top  of  descent 

0.67* 

0.58 

Approach 

0.59 

0.74 

Landing 

0.41 

0.74 

r(15)  =  .606* 
‘Significant  p<'.01 
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HEART  RATE  flNTER-BEAT  INTERVAL) 

Interbeat  interval  discriminated  between  the  low  and  high  workload  flights, 
f  (1,1 6) =27.74,  (MSe=2763,  p<.01)  (Figure  1. 22.2-2  and  Table  1.2.22-2).  A  workload 
by  phase  of  flight  interaction  was  significant  as  well,  F(6, 96) =5.69,  (MSe=446,p<.01). 

There  is  a  strong  trend  for  a  main  effect  of  session,  day  1  has  smaller  I  Bis  than  day  2, 
F(1 ,16)  =  7.81 ,  (MSe =34357,  p<.02).  This  effect  indicates  that  the  pilots  do  not 
experience  as  much  workload  on  the  second  session.  As  was  mentioned  for  the  same 
interaction  found  for  eyeblink,  there  may  be  instability  of  the  measure  over  time.  A  more 
likely  explanation  for  the  interaction  is  that  there  is  adaptation  occurring  because  the  pilot 
is  experiencing  identical  conditions  during  the  retest. 

A  significant  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,96)  =  23.32, 
(MSe  =  1279,  p<.01).  Separate  oneway  ANOVAs  were  performed  and  significant  main 
effects  were  found  for  phase  of  flight  discrimination  for  both  the  low  and  high  workload 
flights,  F(6, 102)  =  22.61,  (MSe=397,  p<.01)  and  f (6, 90)  =  15.04,  (MSe=454,  p<.01), 
respectively.  Inter-beat  interval  could  discriminate  14  out  of  21 ,  and  9  out  of  21 ,  phase  of 
flight  comparisons  for  the  low  and  high  workload  conditions,  respectively. 

In  evaluating  test-retest  reliability  for  inter-beat  interval  6,  out  of  a  possible  14, 
correlations  were  significant  (Table  7.2.2.2-4). 

Examining  inter-rater  reliability  found  78%  of  the  subjects  correlated  significantly  with  the 
means  per  measurement  window. 

HEART  RATE  VARIABILITY  (IBj  STANDARD  DEVIATION) 

Heart  rate  variability  (IBI  standard  deviation)  was  not  able  to  discriminate  the  difference 
between  low  and  high  workload  conditions,  F(1, 16)  =  1.75,  ns  (Figure  7222-3  and  Table 
72.22-5). 

A  significant  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,96)  =  14.13, 
(MSe  =  196,  p<.01).  A  oneway  ANOVA  found  no  main  effect  for  phase  of  flight 
discrimination  for  the  low  workload  condition,  F(6, 102) =2.11,  ns.  A  significant  main 
effect  for  phase  of  flight  discrimination  was  found  for  the  high  workload  flight, 
F(6,90)  =  1 1 .90,  (MSe=63,  p<.01).  Inter-beat  interval  standard  deviation  could 
discriminate  10  out  of  21  phase  of  flight  comparisons  for  the  high  workload  conditions. 

Test-retest  reliability  for  inter-beat  interval  variability  was  not  as  high  as  mean  IBI,  5  out  of 
a  possible  14,  correlations  were  significant  (Table  7.22.2-6). 

In  assessing  inter-rater  reliability  it  was  found  that  44%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  measurement  windows. 

POWER  SPECTRAL  ANALYSIS  fBLOOD  PRESSURE  COMPONENTS 

The  blood  pressure  component  was  not  able  to  discriminate  the  difference  between  low 
and  high  workload,  F(1, 15)  =  4.75,  ns  (Figure  7.2.2.2-4  and  Table  72.22-7). 

A  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,90)=8.54,  (MSe=54, 
p<.01).  Separate  oneway  ANOVAs  were  performed  and  significant  main  effects  were 
found  for  phase  of  flight  discrimination  for  both  the  low  and  high  workload  flights, 
F(6, 102)  =  7.26,  (MSe  =  23,  p<.01)  and  f (6, 90) =3.59,  (MSe=23,  p<.01),  respectively. 
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Figure  7. 2. 2. 2-2 
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Table 

7. 2. 2. 2-3 

Inter-Beat  Interval  (Msec) 

Part  Task  Simulation  Data 

Means  and  Standard  Deviations 

Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

682 

(74.10) 

677 

(72.04) 

Climb 

717 

(81.65) 

712 

(75.89) 

Top  of  climb  714 

(73.84) 

691 

(70.92) 

Cruise 

736 

(70.15) 

710 

(70.42) 

Top  of  descent  715 

(78.61) 

698 

(65.92) 

Approach 

708 

(74.87) 

672 

(66.63) 

Landing 

670 

(61.58) 

656 

(58.52) 

Mi  1C?  19  K4do 
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Table  7. 2. 2. 2-4 


Inter-Beat  Interval  (Msec) 

Part  Task  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.61* 

0.68 

Climb 

0.61* 

0.53 

Top  of  climb 

0.66* 

0.5 

Cruise 

0.53 

0.45 

Top  of  descent 

0.67* 

r(15)  =  0.53 

Approach 

0.67* 

0.55 

Landing 

0.5 

0.53 

r(  16)  =  0.590* 

*  Significant  p  <  0.01 


Figure  7. 2. 2. 2-3 
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Table  7. 2. 2. 2-5 

Inter-Beat  Interval 
Standard  Deviation  (Msec) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

41.58 

(17.05) 

39.04 

(15.09) 

Climb 

41.55 

(13.90) 

43.99 

(10.28) 

Top  of  climb 

47.27 

(16.93) 

50.08 

(12.83) 

Cruise 

43.80 

(9.73) 

49.09 

(14.26) 

Top  of  descent 

47.03 

(11.67) 

50.20 

(12.15) 

Approach 

57.16 

(50.45) 

49.02 

(15.93) 

Landing 

58.54 

(13.80) 

61.33 

(12.82) 

72 


Table  7. 2. 2. 2-6 


Inter-Beat  Interval 
Standard  Deviation  (Msec) 

Part  Task  Simulation 
Test-Retest 

Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.52 

0.7* 

Climb 

0.39 

0.24 

Top  of  climb 

0.61* 

0.31 

Cruise 

0.16 

0.7* 

Top  of  descent 

0.68* 

r(15)  =  0.48 

Approach 

0.78* 

0.52 

Landing 

0.25 

0.16 

r(16)  =  .590* 
‘Significant  p<  .01 


Figure  7. 2. 2. 2-4 


Power  Spectral  Analysis 
(Blood  Pressure  Component) 

Part  Task  Simulation 
Mean  Scores 


Power 

spectral 

analysis 

blood 

pressure 

component 

(Msec)2 


Low  workload 
High  workload 


Takeoff  Climb  Topol  Cruise  Top  of  Approach  Landing 
climb  descent 

(TOC)  (TOD) 


Phase  of  flight 


Table  7. 2. 2. 2-7 

Power  Spectral  Analysis  (Blood  Pressure  Component) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

7.87 

(6.83) 

7.50 

(8.90) 

Climb 

7.14 

(4.72) 

8.38 

(4.94) 

Top  of  climb 

12.92 

(9.77) 

12.37 

(6.63) 

Cruise 

14.89 

(9.21) 

13.72 

(9.05) 

Top  of  descent 

13.91 

(8.65) 

11.95 

(6.61) 

Approach 

12.84 

(8.89) 

10.72 

(5.82) 

Landing 

9.93 

(7.03) 

9.33 

(5.08) 

HUG?  21  H3tH> 


The  Blood  Pressure  component  could  discriminate  7  out  of  21,  and  1  out  of  21,  phase  of 
flight  comparisons  for  the  low  and  high  workload  conditions,  respectively. 

The  blood  pressure  component  demonstrated  poor  test/retest  reliability,  2  out  of  14 
possible  correlations  were  significant  (Table  7.2. 2.2-8). 

In  assessing  inter-rater  reliability  it  was  found  that  17%  of  the  subjects  significantly 
correlated  with  the  means  for  the  measurement  windows. 

POWER  SPECTRAL  ANALYSIS  (RESPIRATION  COMPONENTS 

The  respiration  component  was  able  to  discriminate  the  low  and  high  workload 
conditions  F(1, 15)  =9. 17,  (MSe  =  10,p<.01)  (Figure  7.2.2.2-5  and  Table  7.2.2.2-9). 

A  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,90)=3.C1,  (MSe  =  16, 
p<.01).  The  respiration  component  of  the  power  spectral  analysis  could  not 
discriminate  among  the  phases  of  flight  for  the  low  workload  condition,  F(6,102)  =  2.19, 
ns.  A  significant  main  effect  for  phase  of  flight  discrimination  was  found  for  the  high 
workload  flight,  F(6,90)  =3.43  (MSe  =  7,  p  <  .01 ).  Newman-Kuels  analyses  of  the  windows 
found  no  ability  to  discriminate  the  various  phases  of  flight  for  either  the  low  or  high 
workload  conditions. 

Evaluating  test/retest  reliability  for  the  respiration  component  found  4,  out  of  a  possible 
14,  correlations  significant  (Table  7.2.2.2-10). 

In  assessing  inter-rater  reliability  measures  it  was  found  28%  of  the  subjects  significantly 
correlated  with  the  means  for  the  measurement  windows. 

7.2.2.3  PERFORMANCE  MEASURES 

Control  activity  during  manual  flight  path  control  demonstrated  a  strong  trend  for 
discriminability  between  low  and  high  levels  of  workload.  Tremendous  attrition  of  control 
activity  data  occurred  in  the  low  workload  condition  due  to  autopilot  usage.  During  auto¬ 
flight  control  activity  is  not  considered  a  measure  of  workload  because  the  pilot  is  no 
longer  in  the  control  loop.  Test/retest  reliability  was  high  for  the  control  activity 
measures. 

CONTROL  INPUT  ACTIVITY 

WHEEL  (AILERON1  CONTROL  ACTIVITY 

For  wheel  control  activity  (aileron  inputs)  there  is  a  strong  indication  of  more  control 
activity  for  the  high  workload  condition,  F(1,1)  =  153.2,  ns,  (Figure  7.2.2.3-1  and  Table 
7.2. 2. 3-1). 

A  strong  trend  for  a  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,6)  =  7.83, 
(MSe =68,  p<.012).  No  significant  main  effect  for  phase  of  flight  discrimination  was 
found  for  the  low  workload  flight,  F(6,6) =3.29,  ns.  A  significant  main  effect  was  found  for 
phase  of  flight  discrimination  for  the  high  workload  flight  F(6, 96) =20.45  (MSe=41.7, 
p<.01).  Wheel  control  activity  could  discriminate  10  out  of  21  phase  of  flight 
comparisons  for  the  high  workload  condition. 

Test-retest  reliability  for  wheel  control  activity  was  very  high,  9  significant  correlations  out 
of  a  possible  14  (Table  7.2.2.3-2). 
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Table  7. 2. 2. 2-8 


Power  Spectrum  Analysis 
(Blood  Pressure  Component) 

Part  Task  Simulation 
Test-Retest 

Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.29 

0.57 

Climb 

0.54 

0.34 

Top  of  climb 

0.59 

0.11 

Cruise 

0.31 

0.22 

Top  of  descent  0.73* 

r(15) 

=  0.27 

Approach 

0.24 

0.45 

Landing 

0.5 

r(  15) 

=  0.12 

r(  16)  =  .590* 

‘Significant  p-  .01 
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Figure  7. 2. 2. 2-5 

Power  Spectral  Analysis 
(Respiration  Component) 
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Table  7. 2. 2. 2-9 

Power  Spectral  Analysis  (Respiration  Component) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window  Low-workload  flight  High-workload  flight 


Mean 

SD 

Mean 

SD 

Takeoff 

1.67 

(1.71) 

1.25 

(0.90) 

Climb 

1.90 

(1.80) 

1.85 

(1-35) 

Top  of  climb 

2.15 

(1.88) 

3.62 

(3-07) 

Cruise 

1.77 

(0.99) 

3.07 

(2.43) 

Top  of  descent 

2.16 

(1-77) 

4.09 

(5.22) 

Approach 

2.95 

.  (4-11) 

4.64 

(5.87) 

Landing 

3.26 

(2.37) 

4.10 

(2.84) 

Ht  167  22  R400 
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Table  7.2.2.2-10 


Power  Spectral  Power 
(Respiration  Component) 

Part  Task  Simulation 
Test-Retest 

Reliability  Correlations 


Window 

Low 

High 

Takeoff 

0.17 

0.33 

Climb 

0.33 

0.12 

Top  of  climb 

0.79* 

0.43 

Cruise 

0.02 

0.36 

Top  of  descent 

0.58 

r(15) 

=  0.83 

Approach 

0.93* 

0.79 

Landing 

0.52 

r(  15) 

=  0.23 

r(  16)  =  .590* 
‘Significant  p-  01 
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Figure  7. 2. 2. 3-1 

Wheel  (Aileron)  Control  Inputs  (per  Minute) 

Part  Task  Simulation 
Mean  Scores 


40 

35 

30 

Wheel 

25 

control 

20 

inputs/ 

min 

15 

10 

- Low  workload 

- High  workload 


Takeoff  Climb  Top  of  Cruise  Topol  Approach  Landing 
climb  descent 

(TOC)  (TOD) 

Phase  ol  flight 


Table  7. 2. 2. 3-1 

Wheel  (Aileron)  Control  Inputs 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

8.98 

(2.23) 

9.43 

(6.42) 

Climb 

18.28 

(6.73) 

21.74 

(12.93) 

Top  of  climb 

14.26 

(3.19) 

22.97 

(11.60) 

Cruise 

5.40 

(1.13) 

19.42  . 

(12.52) 

Top  of  descent 

9.95 

(1.35) 

24.70 

(11.06) 

Approach 

21.12 

.  (11.91) 

31.23 

(12.09) 

Landing 

27.64 

(16.83) 

35.79 

(12.98) 

mi67  2?  M*oo 
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Table  7. 2. 2. 3-2 


Wheel  (Aileron)  Control  Inputs 

Part  Task  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

r(  16)  =  0.61 

r(  16)  =0.65 

Climb 

r(16)=  0.58 

r(16)  =0.75 

Top  of  climb 

r(5)  =-0.44 

r(16)=0.77 

Cruise 

— 

r(16)=0.88 

Top  of  descent 

r(  10)  =  0.07 

r(  15)  =0.95 

Approach 

r(12)=  0.86* 

r(16)  =0.70 

Landing 

r(16)=  0.60* 

r(16)  =0.65 

‘Significant 
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Inter-rater  reliability  is  not  available  owing  to  missing  data  for  the  low  workload  conditions 
in  which  the  autopilot  is  used. 

COLUMN  (ELEVATOR)  CONTROL  ACTIVITY 

For  column  control  activity  (elevator  inputs)  there  is  a  strong  indication  for  more  control 
activity  for  the  high  workload  condition,  F(1,1)  =  18.06,  ns,  (Figure  7.2.2.3-2  and  Table 
7.2.2. 3-3). 

A  main  effect  was  found  for  session,  F(1,1)=6782,  (MSe=0.02,  pc. 01).  The  pilots  did 
not  make  as  many  inputs  to  the  elevator  when  flying  the  scenarios  on  the  second  day. 
This  finding  probably  reflects  the  fact  that  the  pilots  became  more  familiar  with  the 
handling  qualities  of  the  motion  base  simulator  over  time. 

A  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,6)  =  14.73,  (MSe  =  146, 
pc. 01).  Separate  oneway  ANOVAs  were  performed  and  significant  main  effects  were 
found  for  phase  of  flight  discrimination  for  both  the  low  and  high  workload  flights, 
F(6,6)  =  24.75  (MSe  =  12,  pc.01)  and  F(6,96)  =  16.30  (MSe=94,  pc.01),  respectively. 
Column  control  activity  could  discriminate  6  out  of  21,  and  9  out  of  21,  phase  of  flight 
comparisons  for  the  low  and  high  workload  conditions,  respectively. 

Test-retest  reliability  for  wheel  control  activity  was  very  high,  10  significant  correlations 
out  of  a  possible  14  (Table  7.2.2.3-4). 

Inter-rater  reliability  is  not  available  owing  to  missing  data  for  the  low  workload  conditions 
in  which  the  autopilot  is  used. 

PEDAL  fRUDDEm  CONTROL  INPUT  ACTIVITY 

When  examining  pedal  control  input  activity  it  should  be  considered  that  the  rudder 
pedals  are  seldom  used  except  during  the  takeoff  and  landing  phases  of  flight.  The 
Climb,  Top  of  Climb,  Cruise,  Top  of  Descent,  and  Descent  measurement  periods 
average  very  few  rudder  inputs,  less  than  zero,  per  minute. 

For  pedal  control  activity  (rudder  inputs)  there  is  no  discernable  difference  between  low 
and  high  workload  conditions,  F(1 , 1  )  =  1 .01 ,  ns,  (Figure  7.2.2.3-3  and  Table  7.2.2.3-5). 

A  main  effect  for  phase  of  flight  discrimination  was  found,  F(6,6)  =  14.73,  (MSe  =  146, 
pc.01).  Separate  oneway  ANOVAs  were  performed  and  significant  main  effects  were 
found  for  phase  of  flight  discrimination  for  both  the  low  and  high  workload  flights, 
f(6,6)  =  16.79  (MSe=26,  pc.01)  and  £(6,96)  =  100.65  (MSe=27,  pc.01),  respectively. 
Pedal  control  activity  could  discriminate  10  out  of  21,  and  11  out  of  21,  phase  of  flight 
comparisons  for  the  low  and  high  workload  conditions,  respectively. 

Test-retest  reliability  for  pedal  control  activity  was  not  as  high  as  either  wheel  or  column 
control  activity,  6  significant  correlations  out  of  a  possible  14  (Table  7.2.2.3-6). 

Inter-rater  reliability  is  not  available  because  of  missing  data  for  the  low  workload 
conditions  in  which  the  autopilot  is  used. 

SECONDARY  TASK 

There  were  some  implementation  problems  encountered  with  the  secondary  task.  A 
post  hoc  examination  of  the  accuracy  of  probe  delivery  indicates  that  ATC  personnel  did 
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Figure  7. 2. 2. 3-2 
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Table  7. 2. 2. 3-3 

Column  (Elevator)  Control  Inputs 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

8.85 

(7-34) 

10.99 

(5.07) 

Climb 

6.76 

(3.19) 

20.20 

(16.67) 

Top  of  climb 

9.38 

(3.72) 

23.66 

(16.47) 

Cruise 

1.50 

(0.99) 

18.06 

(10.55) 

Top  of  descent 

4.74 

(0.09) 

20.02 

(16.29) 

Approach 

13.34 

(4.51) 

29.77 

(17.86) 

Landing 

38.13 

(8.48) 

40.61 

(12.12) 

MU$7?4  R3ob 
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Table  7. 2. 2. 3-4 


Column  (Elevator)  Control  Inputs 

Window 

Part  Task  Simulation 

Test-Retest 
Reliability  Correlations 

Low 

High 

Takeoff 

r(16)  =  0.64* 

r(16)  =0.81* 

Climb 

r(16)  =  0.61* 

r(16)  =0.78* 

Top  of  climb 

.  r{5)  =0.54 

r(16)  =0.83* 

Cruise 

— 

r(  16)  =0.54 

Top  of  descent 

r(  10)  =  0.20 

r(  15)  =0.89* 

Approach 

r(12)  =  0.74* 

r(16)  =0.77* 

Landing 

r(16)  =  0.63* 

r(16)  =0.73* 

‘Significant 

Ft  167  60  H7rk 
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Figure  7.2. 2. 3-3 


Pedal  (Rudder)  Control  Inputs  (per  Minute) 
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Table  7. 2. 2. 3-5 

Pedal  (Rudder)  Control  Inputs 


Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 

High-workload  flight 

Mean 

SD 

Mean 

SD 

Takeoff 

33.39 

(14.34) 

28.11 

(13.71) 

Climb 

3.26 

(3.90) 

0.71 

(0.95) 

Top  of  climb 

1.00 

(1.06) 

1.79 

(1.89) 

Cruise 

0.00 

(0.00) 

0.11 

(0.30) 

Top  of  descent 

0.34 

(0.48) 

0.83 

(1.57) 

Approach 

3.13 

(4.42) 

3.62 

(6.32) 

Landing 

29.98 

(0.23) 

44.69 

(13.58) 

M  \  167  2b  H3dO 
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Table  7. 2. 2. 3-6 


Pedal  (Rudder)  Control  Inputs 

Part  Task  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

r(16)=  0.61* 

CM 

r-~ 

o 

II 

CO 

k — 

Climb 

to 

o 

o 

1 

II 

CO 

CO 

r- 

o 

II 

CO 

Top  of  climb 

— * 

tn 

II 

o 

05 

CM 

o 

II 

CO 

t — 

Cruise 

r(16)=  0 

Top  of  descent 

o 

II 

o 

r(  1 5)  =  0.88 

Approach 

r(12)=  0.99* 

r(  1 6)  =  0.99 

Landing 

r(16)=  0.25 

r(16)=  0.53 

*Significant 


r  1167  61  R7* 


not  deliver  the  probes  in  a  consistent  fashion  across  the  different  subjects. 


SECONDARY  TASK  (RESPONSE  TIME1 

Owing  to  the  case-wise  deletion  of  data  with  missing  cell  entries,  there  were  no  valid 
cases  left  to  analyze  for  testing  validity  (discriminability  of  low/high  workload  or  phase  of 
flight).  Even  with  all  the  flaws  in  the  data  collection  there  was  a  pattern  emerged  for 
faster  response  times  in  the  low  workload  condition  (Figure  7.2.2.3-4  and  Table  7.2.2.3- 
7)- 

A  statistical  analysis  of  the  reliability  measures  for  response  time  was  muddled  because 
of  the  discrepant  sample  sizes  due  to  the  flawed  probe  presentation  (Table  7.2.2.3-8). 

SECONDARY  TASK  (PROBE  ACCURACY) 

No  clear  trends  for  discriminating  low  and  high  workload  are  found  for  probe  accuracy 
(Figure  7.2.2.3-5  and  Table  7.2.2.3-9).  From  Top  of  Climb  through  landing  there  is 
greater  probe  accuracy  for  the  low  workload  condition.  Again,  analyses  for  phase  of 
flight  discriminability  was  precluded  due  to  missing  data  among  the  various  windows. 

Similar  to  Response  Time,  the  statistical  analysis  of  reliability  for  Probe  Accuracy  was 
muddled  because  of  the  discrepant  sample  sizes  for  the  different  phases  of  flight  (Table 

7.2.2.3- 10). 

7.3  RELATIONSHIP  OF  WORKLOAD  MEASURES 

A  correlation  matrix  of  all  the  workload  measures  was  computed  with  the  means  for  the 
14  phases  of  flight  (7  windows  from  the  low  and  high  workload  flights).  Caution  should 
be  exarcised  in  extrapolating  any  relationship  between  the  workload  measures  from  a 
correlation  matrix  constructed  in  this  fashion.  The  correlation  matrix  (Tables  7.3-1  and 

7.3- 2)  allows  the  reader  to  compare  various  measures  to  determine  which  measures  are 
sensitive  to  the  same  changes  in  task  demands. 

The  correlations  (test/retest  and  inter-rater)  presented  earlier  were  constructed  in  an 
entirely  different  fashion.  For  each  workload  score,  the  test/retest  correlations  were 
constructed  individually  for  each  phase  of  flight  by  correlating  the  session  one  and  two 
scores  for  all  the  pilots.  For  each  workload  score,  the  inter-rater  correlations  were 
computed  for  each  pilot  by  correlating  the  average  of  his  session  one  and  two  scores  for 
all  14  phases  of  flight  to  the  group  averages  for  the  14  phases  of  flight. 

Included  in  the  correlation  matrix  are  the  results  from  the  Boeing  TLA.  The  reader 
should  be  aware  of  the  difference  between  the  workload  measures  collected  in  the  Part- 
Task  simulation  and  values  from  the  Boeing  TLA.  The  workload  measures  collected 
from  the  pilots  represents  averages  calculated  from  a  distribution  of  scores  based  on 
sample  sizes  ranging  from  9  to  18.  The  mean  for  each  cell  is  then  used  for  the  various 
workload  measures  in  the  correlation  matrix.  On  the  other  hand,  the  Timeline  Analysis 
represents  a  micro-motion  analysis  which  yields  a  single  value  for  each  body  channel  for 
the  various  phases  of  flight.  It  is  the  single  value  for  TLA  body  channel  which  is  used 
along  with  the  means  for  the  various  workload  measures  to  compute  the  correlation 
matrix. 

7.4  PRINCIPAL  COMPONENT  ANALYSIS 

A  Principal  Component  Analysis  was  computed  on  the  averages  for  the  14  phases  of 
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Figure  7. 2. 2.3-4 
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Table  7. 2. 2. 3-7 

Secondary  Task  (Response  Time) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Low-workload  flight 
Mean  SD 

High-workload  flight 
Mean  SD 

Takeoff 

2.15 

(0.52) 

2.29 

(0.89) 

Climb 

2.71 

(1.42) 

2.93 

(0-97) 

Top  of  climb 

3.09 

(1.51) 

3.19 

(1-28) 

Cruise 

3.15 

(1.51) 

3.39 

(1-79) 

Top  of  descent 

2.69 

(0.87) 

2.84 

(0-73) 

Approach 

2.74 

-(1.22) 

3.25 

(1.51) 

Landing 

2.73 

(1.00) 

3.29 

(1.64) 

H 1 26  R5 
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Table  7. 2. 2. 3-8 


Secondary  Task  (Response  Time) 

Part  Task  Simulation 
Test- Retest 
Reliability  Correlations 


Window 

Low 

High 

Takeoff 

— 

— 

Climb 

r{  1)  =  0.26 

— 

Top  of  climb 

r(5)  =  0.37 

r(6)  =  0.02 

Cruise 

r(4)  =  0.90 

r(2)  =  0.98 

Top  of  descent 

r(  16)  =  0.39 

r(  1 5)  =  0.27 

Approach 

r(13)  =  0.65* 

r(10)  =  0.62 

Landing 

r(3)  =-0.21 

— 

^Significant 

f  1 167  87  RGffc 
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Figure  7. 2. 2. 3-5 

Secondary  Task 

(Probe  Accuracy  -  Percent  Correct) 

Part  Task  Simulation 
>%  r-  Mean  Scores 

)%  —  - Low  workload 

)%  —  / - ~”X  /  - High  workload 


Percent 
correct  60% 
50% 


/  \  \ 
/  \ 

/  \ 

/  \ 


Takeoff  Climb  Top  of  Cruise  Top  of  Approach  Landing 
climb  descent 

(TOC)  (TOD) 


Phase  of  flight 


Table  7. 2. 2. 3-9 

Secondary  Task 

(Probe  Accuracy  -  Percent  Correct) 

Part  Task  Simulation  Data 
Means  and  Standard  Deviations 


Window 


Low-workload  flight 


High-workload  flight 


Mean 

SD 

Mean 

SD 

Takeoff 

60% 

(55%) 

75% 

(45%) 

Climb 

41% 

(44%) 

46% 

(49%) 

Top  of  climb 

61% 

(34%) 

41% 

(36%) 

Cruise 

81% 

(23%) 

55% 

(37%) 

Top  of  descent 

84% 

(15%) 

77% 

(18%) 

Approach 

67% 

'  (27%) 

54% 

(30%) 

Landing 

82% 

(39%) 

44% 

(51%) 

HHO???  H3dt> 
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Table  7. 2. 2. 3-1 


Secondary  Task  (Probe  Accuracy) 

Part  Task  Simulation 

Test-Retest 

Reliability  Correlations 

Window 

Low 

High 

Takeoff 

— 

— 

Climb 

r(1)  =-0.10 

— 

Top  of  climb 

r(5)  =  0.35 

r(6)  =  0.28 

Cruise 

r(4)  =0.16 

r(2)  =  0.74 

Top  of  descent 

r(  16)  =  0.51 

r(  1 8)  =  0.04 

Approach 

r(13)  =  0.60 

r(10)  =  -0.06 

Landing 

f  11G7  00  H4fS 
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Correlation  Matrix 

Part-Task  Simulation 


Table  7.3-1 
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Critical  correlation  values  are  r(12)  =  0.66  or  r(  12)  =  -0.66  are  in  bold 


Correlation  Matrix 

Part-Task  Simulation 


Table  7.3-2 
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flight  (7  each  from  the  low  and  high  workload  conditions)  in  order  to  determine  common 
"underlying"  dimensions  among  the  workload  measures. 

Principal  Component  Analysis  is  a  form  of  Factor  Analysis  that  maximizes  the  variance 
accounted  for  in  the  solution.  Principal  Component  Analysis  (PCA)  sorts  the  measures 
into  factor  loadings  by  maximizing  the  amount  of  variation  that  can  be  explained  by  each 
factor,  and  then  calculates  a  "loading"  for  each  variable  on  the  respective  factors.  The 
loading  is  a  score  between  +1  and  -1,  similar  to  a  correlation  coefficient,  where  the 
absolute  value  indicates  the  strength  of  the  loading  and  the  sign  (plus  or  minus) 
indicates  the  direction  of  the  relationship. 

The  "naming"  of  the  factors  which  emerge  from  any  Factor  Analysis  can  be  the  subject  of 
much  debate.  The  investigators  in  this  project  developed  the  following  labels: 

FACTOR  1  -  Gross  Motor  Activity. 

FACTOR  2  -  Psycho-motor  Activity  (i.e.,  Flight  Path  Control). 

FACTOR  3  -  Cognitive  Activity  (i.e.,  Operation  and  Monitoring  of  Aircraft  Engines 
and  Systems). 

FACTOR  4  -  Mediational  Activity  (i.e.,  Command  Decisions). 

When  selecting  measures  for  a  aircraft  certification  program  measures  should  be 
selected  that  represent  various  FACTORS,  as  opposed  to  selecting  measures  from 
various  domains  (i.e.,  Subjective,  Physiological,  and  Performance).  In  that  way  the 
various  components  of  workload  could  be  quantitatively  assessed,  otherwise  two 
measures  from  various  domains  which  assess  the  same  underlying  dimension  might  be 
brought  to  bear  in  the  assessment  effort. 

Caution  should  be  exercised  when  evaluating  the  PCA  table  because  the  variables 
included  in  the  analysis  has  an  effect  on  the  factors  and  the  loadings.  If  variables  were  to 
be  left  out  of  the  analysis  it  is  likely  that  the  factor  loading  scores  would  change  for  the 
variables. 

The  PCA  table  that  follows  shows  which  measures  load  on  common  dimensions  (Table 
7.4-1).  Factor  loadings  less  than  plus  or  minus  0.25  are  deleted  to  ease  in  the  reading  of 
the  PCA  table. 

7.5  DISCUSSION 

The  intent  of  our  program  is  to  examine  existing  workload  measures  to  determine  the 
validity  and  reliability  of  the  application  of  these  measures  to  new  aircraft  certification 
under  FAR  25.1523.  In  pursuit  of  the  stated  goal  each  workload  measure  has  been 
treafed  as  if  it  were  the  only  measure  being  examined  in  this  project.  There  are  two 
reasons  why  we  are  using  this  approach.  First,  it  was  never  our  goal  to  develop  a 
battery  of  workload  measures  to  be  utilized  in  commercial  aircraft  certification.  It  was  our 
intent  to  subject  each  of  the  candidate  workload  measures  to  the  rigors  of  full  fidelity 
simulation  testing  in  order  to  examine  the  constructs  of  validity  and  reliability.  Second, 
the  measures  are  examined  individually  because  this  is  conceptually  the  only  practical 
manner  to  assimilate  all  the  data  presented. 

In  order  to  provide  quick  look  summaries  of  the  data  two  tables  have  been  compiled. 
The  first  table  summarizes  the  empirical  findings  of  validity  and  reliability  (Table  7.5-1). 
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Table  7.4-1 


Principal  Component  Analysis 

Part-Task  Simulation  Data 


Sorted  rotated  factor  loadings  (pattern) 


Factor  1 

Factor  2 

Factor  3 

Factor  4 

Visual  (TLA) 

0.967 

Blood  pres. 

-0.960 

Manual  left  (TLA) 

0.864 

Average  IBI 

-0.626 

-0.538 

-0.409 

Cognition  (TLA) 

0.623 

-0.401 

0.488 

-0.425 

Manual  right  (TLA) 

0.565 

0.329 

0.456 

Wheel  control  input 

0.281 

0.898 

0.287 

Respiration 

0.866 

0.301 

Stick  control  input 

0.371 

0.845 

IBI  variability 

0.774 

0.338 

Sec.  task  react,  time 

-0.485 

0.663 

0.293 

NASA  task  load  index 

0.388 

0.892 

Overall  workload  score 

0.439 

0.870 

SWAT 

0.498 

0.843 

Verbal  (TLA) 

0.795 

-0.406 

Eye  blink 

0.898 

Pedal  control  input 

0.646 

0.709 

Audition  (TLA) 

0.426 

0.456 

-0.624 

Sec.  task  percent  cor. 

-0.385 

-0.464 

0.397 

VP 

4.914 

4.666 

4.115 

2.610 

The  above  (actor  loading  matrix  has  been  rearranged  so  that  tha  columns  appear  in  decreasing  order 
of  variance  explained  by  the  factors.  The  rows  have  been  rearranged  so  that  for  each  successive 
factor,  loadings  greater  than  0.500  appear  first.  Loadings  less  than  0.25  have  been  blanked. 
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Table  7.5-1 
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The  second  table  presents  the  rank  ordering  of  the  phases  of  flight  for  each  workload 
measure  (Tables  7.5-2  and  7.5-3). 

7.5.1  DISCUSSION  OF  VALIDITY  AND  RELIABILITY  RESULTS 

SUBJECTIVE 

The  subjective  measures  demonstrate  validity  and  reliability,  although  SWAT  was  weaker 
than  the  other  two  measures  in  terms  of  reliability.  The  authors  feel  that  the  low 
overhead  in  terms  of  implementation  and  data  reduction  for  subjective  measures  add  to 
the  attractiveness  of  utilizing  these  types  of  measures  in  an  aircraft  certification  effort. 

The  Pilot  Subjective  Evaluation  (PSE)  technique  developed  by  Boeing  could  not  be 
utilized  in  the  current  study  due  to  the  baseline  versus  new  aircraft  comparison  inherent 
in  the  PSE.  Although  the  criteria  of  the  present  study  could  not  be  brought  to  bear  on 
the  PSE,  the  dividend  of  identification  of  specific  subsystems  which  are  influenced  by  the 
functions  and  factors  of  FAR  25.1523  Appendix  D,  is  of  enormous  benefit  in  an  aircraft 
certification  effort  (Fadden,  1982;  Fadden  and  Ruggerio,  1987). 

PHYSIOLOGICAL 

The  physiological  measures  were  perhaps  the  most  disappointing  in  terms  of  the  ability 
to  discriminate  levels  of  workload  in  a  reliable  manner. 

Horizontal  EOG  was  collected  in  order  to  analyze  eye  movement.  Eye  Movement  per 
unit  of  time  may  be  a  useful  index  of  workload,  but  resource  limitations  did  not  allow  for  a 
careful  reduction  of  the  horizontal  EOG  data  so  no  analyses  were  conducted.  Eyeblink, 
although  reliable,  could  not  discriminate  the  different  workload  conditions.  The  lack  of  a 
main  effect  for  eyeblink  may  be  due  to  the  different  visual  task  demands  in  the  phases  of 
flight  inherent  in  commercial  transport  aircraft  operations.  Certain  phases  of  flight  can  be 
characterized  by  a  great  deal  of  chart  reading,  others  by  the  intense  scan  of  instruments 
on  an  approach,  yet  others  by  quick  scans  at  various  system  instruments  (both  on  the 
forward  instrument  panel  and  the  flight  engineer's  panel)  to  diagnose  system 
malfunctions. 

Inter-beat  interval  (mean  Heart  Rate)  was  a  fairly  robust  measure  in  terms  of  validity  and 
reliability.  We  are  still  concerned  about  the  ability  to  tease  arousal  and  workload  apart 
when  using  a  measure  such  as  Heart  Rate.  Due  to  the  sensitivity  of  Heart  Rate  to 
arousal  we  feel  the  same  pattern  of  results  that  were  found  for  Heart  Rate  might  be  found 
for  an  observer  riding  in  the  jumpseat  in  the  cockpit.  No  attempt  is  being  made  to 
impugn  the  reputation  of  Heart  Rate  with  regards  to  it's  utility  as  a  workload  measure, 
rather  a  word  of  caution  is  being  advanced. 

Heart  Rate  Variability  (standard  deviation  of  the  Inter-beat  Interval)  was  not  able  to 
discriminate  between  the  low  and  high  workload  conditions.  Test/ retest  reliability  was 
good,  but  the  utility  of  reliable  measure  which  cannot  discriminate  between  low  and  high 
workload  is  questioned. 

The  Blood  Pressure  Component  of  the  power  spectral  analysis  could  not  discriminate 
between  the  low  and  high  workload  conditions.  A  few  of  the  test/retest  reliability 
correlations  were  significant,  but  of  no  real  value  since  the  measure  did  not  demonstrate 
discriminability. 
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Part-Task  Simulation  (Rank  Order  of  Phase  of  Flight) 


Table  7.5-3 
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An  anomalous  finding  for  the  Respiration  Component  of  the  power  spectral  analysis  is 
hard  to  reconcile.  The  a  priori  prediction  for  the  respiration  component  is  that  as 
workload  goes  up  the  power  in  the  respiration  portion  of  the  spectrum  should  go  down, 
prn 

The  finding  that  the  respiration  component  increases  with  increasing  workload  warrants 
careful  examination  of  the  measure  in  the  next  simulation  effort. 

PERFORMANCE 

The  discriminability  and  reliability  of  the  wheel  (aileron)  and  column  (elevator)  control 
input  activity  performance  variables  was  very  good.  A  number  of  aircraft  state  variables 
were  collected  and  analyzed  for  the  Part-Task  simulation  included: 

(a)  RMS  flight  director  error  in  the  lateral  and  vertical  direction, 

(b)  Throttle  activity, 

(c)  Altitude  error  over  the  middle  and  inner  marker  (during  the  approach  and 
landing), 

(d)  Lateral  deviation  from  runway  centerline  (during  the  approach  and 
landing), 

(e)  Localizer  and  glideslope  deviation  for  the  #1  and  #2  receivers. 

Only  the  control  input  activities  demonstrated  any  validity  by  discriminating  between  the 
low  and  high  workload  flights. 

The  lack  of  validity  and  reliability  results  for  the  Secondary  Task  does  not  necessarily 
mean  it  is  a  poor  index  of  workload,  rather  it  points  out  the  unsuitable  nature  of  the 
measure  for  assessing  workload  in  commercial  transport  operations.  In  the 
implementation  of  the  Secondary  Task  in  the  Part-Task  simulation  a  great  deal  of 
secondary  variance  was  generated  for  Response  Time  due  to  the  fact  that  commercial 
pilots  rarely  keep  their  hand  on  the  PTT  switch  on  the  yoke.  Therefore  longer  reaction 
times  were  found  for  relatively  low  periods  of  workload  because  the  pilot  had  to  literally 
reach  for  the  PTT  switch  to  complete  a  response.  In  a  military  aircraft,  particularly  those 
designed  with  a  HOTAS  concept  (hands  on  throttle  and  stick),  the  implementation  of  this 
sort  of  secondary  task  would  not  be  subject  to  the  source  of  secondary  variance  found 
in  the  Part-Task  simulation. 

7.5.2  DISCUSSION  OF  CORRELATION  MATRIX 

The  subjective  measures  demonstrate  a  high  inter-correlation,  indicating  that  they  are 
influenced  by  the  same  changes  in  task  demands. 

The  Blood  Pressure  component's  high  correlation  to  control  input  activity  is  what  would 
be  expected  if  both  measures  are  tapping  some  sort  of  physical  component  of  workload. 

A  summary  of  the  most  notable  findings  from  the  correlation  matrix  include: 

(a)  The  subjective  ratings  are  significantly  inter-correlated  (r=0.97  being  the 
smallest  correlation). 

(b)  The  negative  correlation  of  inter-beat  interval  and  various  indices  of 
physical  workload  (e.g.,  column  &  pedal  control)  is  significant. 

(c)  A  number  of  correlations  approach  significance  between  IBI  and  wheel 
control  inputs,  SWAT,  NASA-TLX,  the  1-to-20  point  overall  workload  score, 
and  the  visual,  manual  left  arid  right  from  the  TLA. 
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(d)  Some  findings  are  unexpected  from  the  examination  of  the  correlation 
matrix.  The  respiration  component  of  the  Power  Spectral  analysis  is 
positively  correlated  with  the  subjective  ratings  and  control  inputs. 
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8.0  WORKSHOP  TWO  (SEATTLE,  WA.) 

8.1  OBJECTIVES 

Workshop  number  two  was  conducted  in  order  to  present  the  results  from  the  Part-Task 
simulation,  and  obtain  recommendations  for  improvements  for  the  Full-Mission 
simulation. 

Participants  were  asked  to  comment  on  the  fidelity  of  the  simulation  scenarios  and  the 
applicability  of  the  candidate  workload  measures  to  aircraft  certification.  The  Full- 
Mission  simulation  test  plan  was  presented  for  review.  Audience  comments  were 
reviewed  and  revisions  to  the  test  plan  were  incorporated  prior  to  the  Full-Mission 
simulation. 

8.2  ATTENDEES 

Eighty  attendees  were  drawn  from  a  wide  cross-section  of  operational  personnel  and 
potential  workload  measurement  users.  University  scientists  from  the  first  workshop 
were  invited  to  attend  to  help  assess  the  scientific  quality  of  the  test  design,  the  results, 
and  the  appropriateness  of  the  conclusions  drawn  from  the  data.  The  remainder  of  the 
attendees  were  from  aerospace  industry,  government  regulatory  agencies,  military 
workload  experts,  and  NASA. 

8.3  DESCRIPTION  OF  EVENTS 

The  first  day  of  the  workshop  was  spent  reviewing  the  methodology  and  results  from  the 
Part-Task  simulation.  The  second  day  was  spent  reviewing  "lessons  learned"  from  the 
Part-Task  simulation  and  appropriate  changes  to  be  incorporated  into  the  Full-Mission 
simulation. 

The  proceedings  of  Workshop  number  two  includes  copies  of  all  the  workshop 
presentations  (Boucek  et.  al.,  1988).  The  data  collected  in  Part-Task  testing  and  the 
results  of  the  Timeline  Analysis  were  included.  In  addition,  a  summary  of  the  discussions 
held  over  the  two  days  was  provided.  It  included  pertinent  points  made  by  the  audience 
that  were  considered  for  the  full-mission  simulation. 

8.4  RESULTS 

Aside  from  presenting  the  results  from  the  Part-task  simulation,  the  most  important 
reason  for  holding  Workshop  Two  was  to  gather  comments  and  suggestions  for 
improvements  on  the  simulation  testing  for  the  Full-mission  simulation.  The  most 
important  topics  are  discussed  below: 

IMPLEMENTATION  OF  A  WORKLOAD  MEASURE 

The  feasibility  of  using  a  decision-tree  based  a  workload  scale,  in  addition  to  the  other 
subjective  measures  taken,  was  discussed.  The  possibility  of  using  a  "Modified  Cooper- 
Harper"  or  the  "Bedford  Scale  "  was  addressed.  The  principle  motivation  was  to  use  a 
subjective  rating  technique  with  empirical  evidence  of  validity  and  reliability.  The  1-to-20 
point  overall  workload  scale  used  in  the  Part-Task  simulation  was  an  extension  of  the 
NASA-TLX  methodology,  and  did  not  possess  an  empirical  record  of  the  other  subjective 
rating  methods. 
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A  number  of  suggestions  were  made  regarding  an  In-Flight  subjective  measure  to 
provide  opportunity  for  a  comparison  of  In-Flight  versus  Post-Flight  subjective 
assessment. 

The  question  of  which  rating  scale  should  be  employed  in  the  simulation  was  discussed. 
SWAT  scores  can  be  taken  verbally  using  verbal  probes  was  the  most  feasible. 
Although  it  is  feasible  to  implement  any  subjective  rating  verbally  during  flight,  SWAT  was 
deemed  the  easiest  due  to  the  smallest  rating  scale  (e.g.,  3  point:  low,  medium,  and 
high).  The  Bedford  and  Modified  Cooper-Harper  scales  both  contain  10  point  rating 
scales,  while  the  NASA-TLX  contains  20  point  rating  scales. 

The  question  of  when  the  ratings  would  be  taken  In-Flight  was  also  raised.  If  the  ratings 
were  requested  at  the  end  of  the  measurement  windows,  and  the  measurement  window 
was  specifically  demarcated,  then  pilots  might  modify  their  flying  performance,  or  in 
some  fashion  affect  the  other  workload  measures  being  collected.  Demarcation  of  the 
measurement  window  had  to  be  avoided  otherwise  the  pilot  might  change  his 
performance  (i.e.,  try  harder)  at  specific  times  because  he  knows  he  is  being  measured. 
It  was  suggested  that  measurements  be  taken  at  variable  intervals,  both  during  and 
outside  of  measurement  windows.  The  problem  with  "random"  probing  is  the  lack  of 
empirical  comparison  available  due  to  data  collected  outside  of  the  measurement 
windows.  It  was  decided  to  collect  data  at  the  end  of  the  measurement  window.  The 
problem  with  the  "instantaneous"  probing  at  the  end  of  the  window  is  that  the  event 
rating  would  reflect  only  the  workload  at  that  particular  moment  and  not  the  workload  of 
the  whole  measurement  period  (i.e.,  the  measurement  window).  The  final  decision  to 
probe  for  an  "instantaneous"  rating  was  based  on  not  wanting  the  In-Flight  probe  to 
artifact  the  other  workload  measures  being  collected  concurrently. 

SECONDARY-TASK  ADMINISTRATION 

It  was  decided  to  eliminate  the  secondary  task  from  the  Full-mission  simulation.  The 
design  of  the  secondary  task  was  discussed  in  length.  Flight  operations  personnel 
questioned  the  implementation  of  the  secondary  task  in  the  flight  test  portion  of 
certification.  It  was  determined,  however,  that  the  secondary  task  may  be  valuable  even 
if  it  can  only  be  used  in  simulation.  Even  though  it  was  handicapped  during  the  part-task 
testing  by  problems  that  were  encountered,  response  time  showed  a  trend  for  workload 
discriminability. 

Too  many  problems  were  encountered  in  the  Part-task  simulation  with  the  secondary 
task.  The  push-to-talk  (PTT)  switch  used  by  the  pilot  to  respond  to  the  positive  probes 
blanked  ATC  with  the  switch  closure.  The  feasibility  of  using  a  different  switch  was 
discussed.  Several  other  types  of  "response  to  a  probe"  task  were  suggested  (e.g., 
squawking  different  transponder  codes)  for  use  as  a  secondary  task  measure.  The 
suggested  tasks  lacked  the  requirements  of  both  positive  and  negative  probes,  and  a 
sufficient  number  of  probes  to  gather  a  good  base  of  data.  Since  data  is  collected  on 
the  Captain  only,  the  task  must  also  be  one  that  is  normally  performed  by  the  Captain. 

Other  aircraft  environments  (e.g.,  tactical)  can  more  readily  adapt  a  secondary  task 
methodology.  The  reaction  to  ground  threats  using  voice  activated  counter-measures 
has  shown  a  great  deal  of  promise  (Vidulich  and  Bortolussi,  1988). 


USE  OF  AUTOPILOT 

For  the  Full-mission  simulation  it  was  decided  to  dispatch  the  aircraft  with  the  autopilot 
INOP  for  all  the  flights  in  order  to  be  able  to  interpret  the  control  input  activity  measures. 
In  Part-Task  simulation  testing,  the  use  of  the  autopilot  in  the  low  workload  condition  was 
left  to  the  pilot's  discretion,  and  thus,  its  use  became  inconsistent  across  the  subject 
population.  It  was  felt  that  tighter  control  on  the  use  of  autopilot  (or  preventing  its  use 
altogether)  in  Full-Mission  testing  should  be  required. 

INCAPACITATION 

In  order  to  manipulate  the  FAR  25.1523  Appendix  D,  factors  of  crew  member  unavailable 
at  crew  station  and  incapacitation  a  feigned  incapacitation  on  the  part  of  the  confederate 
First  Officer  was  discussed.  Discussion  centered  around  the  possibility  of  a  negative 
reaction  by  the  subject  pilot.  It  might  not  be  immediately  obvious  that  the  First  Officer's 
incapacitation  was  part  of  the  simulation  scenario,  and  the  pilot  might  interrupt  the 
simulation  to  seek  aid  for  the  stricken  crew  member. 

In  order  to  selectively  manipulate  the  Captain's  task  demands  similar  to  the  increased 
loading  of  a  First  Officer's  incapacitation  it  was  decided  that  on  one  flight  the  Captain 
would  be  required  to  tune  and  talk  on  the  command  radio. 

WORKLOAD  MEASURES  TO  BE  USED  IN  THE  FULL-MISSION  SIMULATION 

Because  of  inputs  (discussed  in  the  previous  section)  from  the  attendees  at  Workshop 
Two  the  following  measures  were  selected  for  the  Full-Mission  simulation  tests  to  be 
conducted  at  NASA-Ames: 

SUBJECTIVE 

SWAT  (In-flight) 

SWAT  (Post-flight) 

Bedford  (Post-flight) 

PHYSIOLOGICAL 

Heart  Rate  (Inter-beat  interval) 

Heart  Rate  Variability  (Standard  deviation  of  IBI) 

Mulder  Analysis  for  Blood  Pressure 
Mulder  Analysis  for  Respiration 
Eyeblink  Rate 

PERFORMANCE 

Primary  Task 

Control  Inputs 
Wheel 
Column 
Pedals 

Secondary  Task 
None 
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9.0  FULL-MISSION  SIMULATION  TESTING 

Again  the  simulation  was  performed  at  the  NASA-Ames  Research  Center,  Moffett  Field, 
California.  Air-Traffic  Control  was  again  simulated  to  ensure  that  the  fidelity  was  as  close 
to  real-world  conditions  as  possible. 

9.1  METHOD 

Similar  to  the  Part-task  simulation  tests,  only  Captains  were  used  for  the  simulation. 
Again,  no  attempt  was  made  to  address  crew  performance  from  a  cockpit  resource 
management  viewpoint. 

9.1.1  SUBJECTS 
PILOTS 

Sixteen  Airline  Transport  Pilots  (ATP),  (from  American,  United,  TWA,  and  Eastern), 
served  as  subjects  in  the  experiment.  Subjects  were  all  male  ranging  in  age  from  44  to 
58.  Subjects  were  either  currently  FAR  Part  121  qualified  as  Captain  for  the  B-727  or  had 
spent  5  years  of  duty  as  Captain  for  the  B-727. 

FLIGHT  CREW 

Two  confederates  participated  in  the  simulation  study  as  the  First  Officer  and  Flight 
Engineer,  respectively.  Preston  Suit,  Flight  Crew  Training  at  Boeing  Commercial 
Airplanes,  served  as  the  First  Officer  in  the  study.  Preston  also  gave  the  briefing  on 
differences  training  and  the  routes  to  be  flown  in  the  study.  Doranne  VonEnde  and 
Hugh  Campion,  both  qualified  flight  engineers,  served  as  Flight  Engineers  in  the  study. 
Both  the  First  Officer  and  Flight  Engineer  were  cognizant  of  the  workload  manipulations 
a  priori,  and  attempted  to  give  each  pilot  similar  treatment  during  the  simulation. 

9.1.2  EXPERIMENTAL  DESIGN 

The  factors  that  drove  the  design  included: 

a.  Different  levels  of  workload  as  defined  by  the  functions  and  factors 
of  FAR  25. 1 523  Appendix  D. 

b.  Two  test  sessions  in  order  to  evaluate  reliability. 

c.  Sampling  of  various  phases  of  flight  in  order  to  represent  task 
demands  representative  of  operational  conditions. 

9.1. 2.1  INDEPENDENT  VARIABLES 

TEST/RETEST 

Similar  to  the  Part-task  simulation  a  test/retest  methodology  was  employed  to  evaluate 
the  reliability  of  the  workload  measures.  The  period  between  the  two  simulation  test 
periods  was  at  least  27  days,  and  was  as  long  70  days. 

LEVELS  OF  WORKLOAD 

Three  different  flight  scenarios  were  used  to  test  functions  and  factors  of  FAR  25.1523 
Appendix  D,  in  tne  Full-Mission  simulation.  The  two  short  hops  from  the  Part-Task 
simulation  were  carried  over,  as  well  as  a  longer  (1  hour  30  minute)  flight. 


104 


1 


No  effect  was  found  for  route  flown  during  the  Part-Task  simulation.  In  the  present  study 
there  was  no  attempt  made  to  counter-balance  route  (SFO-SCK,  SMF-SFO,  &  LAX-SFO- 
OAK-SMF)  and  workload  conditions  (Nominal,  Communication,  and  Malfunction). 

The  SFO-SCK  flight  is  a  "Nominal"  workload  flight.  The  autopilot  is  MEL,  as  it  is  for  all  the 
flights,  but  otherwise  the  conditions  are  ideal.  An  ILS  approach  and  landing  occurs  at 
SCK. 

The  SMF-SFO  flight  is  the  "Communication"  flight.  In  order  to  simulate  the  duties  of  the 
Pilot  Flying  (PF)  in  a  two  crew  cockpit  in  which  the  PNF  is  occupied  (or  out  of  the  cockpit 
area)  with  other  duties,  the  PF  must  tune  and  talk  on  the  command  radio  from  takeoff 
through  landing.  Other  than  the  "communication"  manipulation  the  copilot  and  flight 
engineer  perform  their  normal  flight  duties.  An  ILS  approach  and  landing  occurs  at  SFO. 

The  LAX-(SFO)-(OAK)-SMF  flight  is  the  "Malfunction"  flight.  During  the  flight  (between 
„  windows  3  and  4)  the  crew  receives  a  message  (company  contact  using  SELCAL)  that 

SFO  has  closed  because  of  a  power  failure,  and  they  are  requested  to  divert  to  OAK. 
The  weather  at  OAK  is  marginal,  going  below  minimums  as  the  crew  approaches  the 
middle  marker.  At  decision  height  the  runway  is  not  visible  and  a  missed  approach  is 
executed.  Passing  2,000  feet  on  the  missed  approach  the  number  one  engine  fails.  One 
<  minute  after  passing  6,800  feet,  during  the  climb  to  7,000  feet,  the  "A"  hydraulic  system 

loses  pressure  and  quantity  until  complete  failure  ensues.  The  crew  continues  the  climb 
to  their  enroute  altitude  of  7,000  feet  for  the  remainder  of  the  flight  to  the  alternate,  SMF. 
The  crew  continues  the  flight  to  an  ILS  approach  and  landing  in  good  weather  at  SMF. 

A  table  is  provided  that  contains  a  summary  of  the  workload  manipulations  in  order  to 
aid  the  reader  in  understanding  the  different  task  demands  for  the  low  and  high  workload 
flights  (Table  9. 1.2. 1-1). 

PHASES  OF  FLIGHT 

Seven  phases  of  flight  were  examined  in  the  simulation  test. 

(a)  Takeoff 

(b)  Climb 

(c)  Top  of  Climb  (TOC) 

(d)  Cruise 

(e)  Top  of  Descent  (TOD) 

(f)  Approach 

#  (g)  Landing  (or  Missed  Approach) 

Eleven  phases  of  flight  were  examined  in  the  malfunction  flight,  the  same  seven  as  listed 
above  plus  the  four  segments  listed  below. 

i  (h)  #1  Engine  Failure 

(i)  "A"  System  Hydraulic 

(j)  Approach 

(k)  Landing 

All  the  flights  contain  seven  measurement  "windows"  in  common,  additionally,  the  long 
flight  contains  an  additional  four  measurement  windows: 
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Full-Mission  Simulation 

Workload  Levels 


Table  9.1 .2.1-1 
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•  Hydraulic  system  “A”  failure 


WINDOW 

OPENING  EVENT 

CLOSING  EVENT 

(a)  TAKEOFF 

E.P.R.  >  1.5 

Flaps  5  degrees 

(b)  CLIMB 

Flaps  up 

1  Minute  later 

(c)  TOP  OF  CLIMB 

10,000  feet 

2  Minutes  later 

(d)  CRUISE 

3  Minutes  after 

10,000  feet 

2  Minutes  later 

(e)  TOP  OF  DESCENT 

Throttles  to  idle 

2  Minutes  later 

(f)  APPROACH 

Localizer 

Activation 

Outer  Marker 

(g)  LANDING 

Middle  Marker 

30  Seconds 

(or  MISSED  APPROACH) 

(for  the  long  flight) 

(h)  ENGINE 

2,000  feet  after 

2  Minutes  later 

FAILURE 

Middle  Marker 

(i)  "A"  SYSTEM 

1  Minute  after 

2  Minutes  later 

HYDRAULIC 

FAILURE 

6,800  feet 

(j)  APPROACH 

Localizer 

Activation 

Outer  Marker 

(k)  LANDING 

Middle  Marker 

30  Seconds  later 

Two  graphics  are  provided  to  illustrate  the  flight  scenarios  pictorially  in  order  to  aid  the 
reader  in  understanding  the  measurement  windows  (Figures  9. 1.2. 1-1  and  9. 1.2. 1-2). 

As  was  the  case  with  the  Part-Task  simulation,  window  and  phase  of  flight  are  used  as 
synonyms.  Window  is  used  when  referring  to  experimental  design  or  measurement,  and 
phase  of  flight  is  used  when  discussing  results. 

Some  changes  were  made  to  the  length  of  the  windows  based  upon  the  testing 
experience  from  the  Part-Task  simulation.  Many  of  the  windows  were  made  a  common 
length  (2  minutes)  to  facilitate  data  reduction.  The  most  notable  change  was  the 
shortening  of  the  landing  window.  In  the  Part-Task  simulation  the  Landing  window  (1 
minute  30  seconds)  often  found  the  pilot  "sitting"  on  the  runway  for  45  seconds  to  1 
minute,  waiting  for  the  measurement  period  to  end.  In  the  Full-Mission  simulation  the 
landing  window  was  shortened  (30  seconds),  and  more  accurately  reflected  only  the 
activities  involved  with  landing  the  aircraft. 

SIMULATION  SCHEDULE 

Again  a  test/retest  method  was  used  to  evaluate  the  reliability  of  the  workload  measures. 
The  period  between  the  two  simulation  test  sessions  was  at  least  three  months  for  every 
subject  tested,  and  much  longer  in  many  bases.  Pilots  flew  three  scenarios,  a  Nominal, 
Communication,  and  Malfunction  flight  on  both  visits.  In  the  present  study  no  attempt 
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NOMINAL  AND  COMMUNICATION  FLIGHTS 


Figure  9. 1 . 2 . 1 -1 
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CC2068.01 


MALFUNCTION  FLIGHT 


Figure  9.1 .2.1-2 
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NO.  1  ENGINE 
FAIL 


was  made  to  counter-balance  route  and  workload  conditions  (Nominal,  Communication, 
and  Malfunction). 

9.1 .2.2  DEPENDENT  VARIABLES 
SUBJECTIVE  RATING 

SWAT  (In-Flight  and  Post-Flight)  and  the  Bedford  rating  scale  (Figure  9.1. 2. 2-1)  were 
used  in  the  Full-Mission  simulation. 

The  In-Flight  implementation  of  SWAT  was  accomplished  by  ATC  probing  the  pilot  for 
event  ratings  at  the  end  of  a  measurement  period.  An  example  of  a  probe  is,  'TWA  241 
give  us  your  TIME,  EFFORT,  and  STRESS  rating."  Pilots  were  instructed  to  give  event 
ratings  for  the  workload  they  were  experiencing  at  the  moment  they  were  probed.  When 
comparing  the  In-Flight  ratings  of  SWAT  to  other  workload  measures  it  should  be  kept  in 
mind  that  In-Flight  SWAT  ratings  reflect  an  instantaneous  assessment  whereas  the  other 
measures  reflect  workload  for  the  entire  measurement  period. 

The  Post-Flight  subjective  ratings,  SWAT  and  Bedford,  were  collected  using  the  same 
method  as  was  employed  in  the  Part-Task  simulation.  A  video  tape  of  the  window  was 
played  for  the  pilot  after  all  the  simulation  runs  for  the  session  were  computed.  When  the 
measurement  window  was  over  the  tape  was  stopped  and  the  pilot  made  their  ratings, 
first  SWAT  and  then  Bedford.  When  the  subject  had  completed  their  rating  of  a  single 
measurement  window  they  were  instructed  to  turn  the  page  of  their  rating  booklet,  and 
asked  not  to  refer  to  previous  ratings. 

PHYSIOLOGICAL  INSTRUMENTATION 

The  physiological  workload  measures  collected  were  the  same  as  those  used  in  the 
Part-task  simulation:  Eyeblink  rate,  Interbeat  Interval,  Standard  Deviation  of  Interbeat 
Interval,  Power  Spectral  Analysis  (Blood  Pressure  and  Respiration  Component). 

PERFORMANCE  DATA 

The  performance  measures  collected  were  the  same  as  those  used  in  the  Part-task 
simulation:  control  input  activity  for  the  wheel,  column,  and  rudder  pedals.  The 
secondary  task  was  dropped  from  the  Full-mission  simulation.  All  performance  data  was 
collected  at  a  rate  of  10  Hertz. 

9.1.3  PROCEDURES  AND  EQUIPMENT 

SIMULATOR 

The  Man-Vehicle  System  Research  Facility's  B-727  simulator  was  again  used  as  the 
testbed  for  the  following  simulation  study. 

SCENARIO 

There  were  three  routes  flown  for  the  Full-Mission  simulation.  The  routes  were 
Sacramento  to  San  Francisco  (SMF-SFO),  San  Francisco  to  Stockton  (SFO-SCK),  and 
Los  Angeles  diverted  from  San  Francisco  to  a  missed  approach  at  Oakland  finally 
landing  in  Sacramento  (LAX-SFO-OAK-SMF).  The  two  short  hop  flights,  (SMF-SFO)  & 
(SFO-SCK)  were  flown  at  1 1 ,000  feet  enroute  to  the  destination,  for  an  ILS  approach  and 
landing.  The  long  flight  (LAX-SFO-OAK-SMF)  is  flown  at  31,000  feet,  enroute  to  San 
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The  Bedford  Pilot  Workload  Rating  Scale 

The  decision  making  process  is  started  at  the  bottom  left  corner  of  the  “decision  tree” 


Figure  9. 1 .2.2-1 
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Francisco  there  is  a  diversion  to  Oakland,  where  a  missed  approach  is  executed  (owing 
to  weather),  followed  by  a  diversion  to  a  landing  at  the  alternate,  Sacramento. 

Subjects  received  the  same  order  of  routes  for  session  one  and  two.  The  presentation 
of  flights  was  counter-balanced  across  pilots  and  is  further  discussed  in  the  test  order 
section.  Subjects  receive  the  same  order  of  routes  during  testing  for  session  one  and 
two.  The  presentation  of  flights  was  counter-balanced  across  pilots. 

9.1.4  STATISTICAL  ANALYSES 

A  nominal  alpha  level  of  .01  was  adopted  for  each  statistical  comparison,  as  was  done  in 
the  Part-task  simulation. 

Predictions  about  the  differences  between  the  various  flights  was  not  confirmed  in  an  a 
priori  manner  using  the  Boeing  Timeline  Analysis  as  was  done  in  the  Part-task 
simulation.  The  three  flights  were  tailored  to  be  extremely  similar,  except  for  the  specific 
manipulation  of  FAR  25.1523  Appendix  D  functions  and  factors.  Discriminability  by  a 
workload  measure  among  the  flights  would  indicate  sensitivity  of  the  measure  to 
manipulations  of  the  functions  and  factors  of  FAR  25.1523  Appendix  D. 

Again,  a  decision  rules  were  established  to  determine  if  a  given  workload  measure  has 
demonstrated  validity  and  reliability.  The  threshold  chosen  could  generate  an  argument 
as  to  the  appropriateness  of  the  threshold,  but  the  rule  is  necessary  for  a  discussion  of 
the  "goodness"  (validity  and  reliability)  of  the  measure. 

9.1. 4.1  VALIDITY  ANALYSES 

For  a  workload  measure  to  demonstrate  validity  it  must  discriminate  among  the  various 
flights  which  manipulate  the  functions  and  factors  of  FAR  25.1523  Appendix  D.  An 
overall  2X3X7  repeated  measures  ANOVA  was  performed  on  each  workload  measure, 
with  the  factors  of  session  (1  or  2),  type  of  workload  (Nominal,  Communication,  or 
Malfunction),  and  phase  of  flight,  respectively.  Separate,  pair-wise,  comparisons  among 
the  flights  were  then  conducted  in  order  to  examine  differences  in  workload  predicted  a 
priori  based  on  different  FAR  25.1523  Appendix  D,  tasks  included  in  the  different  flights. 

The  analyses  of  the  Full-Mission  simulation  have  an  additional  complication  in  that  there 
are  three  levels  of  workload,  as  opposed  to  the  low  and  high  of  the  Part-Task  simulation. 
Just  as  with  the  Part-Task  simulation,  an  interaction  of  workload  and  phase  of  flight 
should  be  present  in  order  to  demonstrate  "selective"  discriminability. 

Oneway  ANOVAs  were  computed  for  each  flight  in  order  to  determine  the  discriminability 
of  phase  of  flight.  If  the  oneway  ANOVA  was  found  to  be  significant  the  Newman-Kuels 
range  statistic  was  applied  to  determine  which  phases  of  flight  were  significantly  different 
from  one  another. 

Case-wise  deletion  still  remains  an  artifact  for  the  repeated  measures  ANOVA.  Any  pilot 
that  has  missing  data  in  any  cell  in  the  analysis  is  dropped  from  the  ANOVA. 

All  significant  F  ratios  are  reported,  and  the  results  of  the  test  for  a  main  effect  of 
workload  are  reported  whether  there  is  a  statistically  significant  finding  or  not. 

The  same  approach  will  be  taken  in  examining  the  discriminability  of  phase  of  flight  by  a 
workload  measure  as  was  done  in  the  Part-Task  simulation.  Utilizing  a  paired- 


112 


comparison  approach  the  various  phases  of  flight  were  compared  to  one  another  using 
the  Newman-Kuels  range  statistic. 

9.1 .4.2  VALIDITY  DECISION  CRITERIA 


For  an  assessment  technique  to  demonstrate  validity  it  should  find  a  main  effect  for 
workload  type.  Similar  to  a  main  effect  of  workload,  an  interaction  of  workload  and 
phase  of  flight  should  be  demonstrated  to  provide  evidence  of  validity  or  a  workload 
measure. 

Similar  to  the  Part-Task  simulation,  no  systematic  attempt  was  made  to  describe  the 
nature  of  the  Newman-Kuels  range  statistic  due  to  complications  arising  from  the  large 
number  of  comparisons.  Our  approach,  simply  stated,  is  the  more  significant 
>  differences  that  were  found  the  more  discriminable  the  workload  measure  was  thought 

to  be. 

9.1. 4.3  RELIABILITY  ANALYSES 

t  Test/retest  was  assessed  using  the  same  method  as  the  Part-Task  simulation.  For  a 

given  workload  measure,  each  pilot's  scores  (session  1  and  2)  are  paired,  then  a 
correlation  is  computed  using  all  the  pilot  score's  for  each  measurement  window.  The 
Nominal  and  Communication  flights  will  yield  seven  correlation  coefficients  each, 
corresponding  to  the  seven  measurement  windows.  The  Malfunction  flight  will  yield 
eleven  correlation  coefficients  corresponding  to  the  eleven  measurement  windows. 

Similar  to  the  Part-Task  simulation,  the  correlation  coefficients  for  test/retest  reliability  are 
based  on  as  many  data  points  as  are  available.  Case-wise  deletion  is  not  performed  on 
pilots  with  missing  data. 

Inter-rater  reliability  was  assessed  in  the  same  fashion  as  the  Part-Task  simulation.  For  a 
given  workload  measure,  the  25  scores  (7  windows  from  Nominal  &  Communication  and 
1 1  windows  the  Malfunction  flight)  for  the  pilot  (average  session  1  and  2  scores)  are 
paired  with  the  group  mean  for  the  respective  measurement  windows  to  yield  individual 
correlation  coefficients.  Inter-rater  reliability  is  then  expressed  as  a  percentage  of  the 
pilots  that  show  a  significant  correlation  with  the  group  mean. 

9.1. 4.4  RELIABILITY  DECISION  CRITERIA 

Similar  to  the  question  of  reliability  criteria  brought  up  in  the  Part-Task  simulation,  the 
exact  criterion  to  determine  whether  or  not  a  workload  measure  is  reliable  can  be  subject 
to  much  debate.  When  a  large  number  of  statistics  are  computed  the  Type  I  error  rate  is 
inflated  proportional  to  the  number  of  coefficients  computed.  Again,  a  comparison-wise 
A  Type  I  error  rate  is  adopted  when  examining  the  correlation  coefficients  because  of  the 

hypotheses  advanced  (namely  that  a  measure  ought  to  provide  the  same  pattern  of 
results  with  repeated  application). 

In  the  present  study  it  was  decided  that  20%  of  the  coefficients,  5  out  a  possible  25, 
correlations  ought  to  be  significant  for  the  workload  metric  to  be  considered  reliable. 
There  is  nothing  magical  about  the  20%  figure,  nor  is  any  consideration  given  to  the 
phase  of  flight  or  workload  conditions  (Nominal,  Communication,  or  Malfunction)  in 
which  the  significant  correlations  are  found. 
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Another  index  of  reliability  is  the  correlation  of  each  pilot's  scores  (average  test/retest)  to 
the  group  mean.  In  order  to  feel  confidence  using  this  index  at  least  half  of  the  subjects 
should  correlate  significantly  with  the  group  mean. 

9.2  RESULTS 

9.2.1  SUMMARY  OF  RESULTS 

SWAT,  the  Bedford  scale,  Heart  Rate  (with  qualification),  and  Control  Inputs  (Wheel, 
Column,  &  Pedal)  demonstrated  evidence  of  validity  by  discriminating  workload,  as  well 
as  finding  a  significant  interaction  of  workload  and  phase  of  flight.  Heart  Rate  did  not  find 
a  significant  main  effect  of  workload,  although  a  strong  trend  did  exist  for  being  able  to 
discriminate  among  the  levels  of  workload. 

SWAT,  the  Bedford  scale,  Eyeblink,  Heart  Rate,  the  Blood  Pressure  Component  of  the 
Power  Spectral  Analysis  and  Control  Inputs  (Wheel,  Column,  &  Pedal)  demonstrated 
evidence  of  reliability  by  finding  at  least  five,  out  of  a  possible  25,  test/retest  correlations 
were  significant.  The  above  listed  measures  also  demonstrated  inter-rater  reliability  by  at 
least  50%  of  the  pilots  significantly  correlated  to  the  group  means  for  the  respective 
workload  conditions. 

9.2.1 .1  SUBJECTIVE  RATINGS 

SWAT  (Post-Flight)  and  the  Bedford  scale  demonstrated  evidence  of  validity  by 
discriminating  among  the  three  different  workload  flights.  SWAT  (In-Flight)  was  not  able 
to  discriminate  among  all  three  flights.  All  three  measures  SWAT  (in  &  Post-Flight)  and 
Bedford  all  demonstrated  evidence  of  reliability. 

SWAT  (In-Flight) 

To  yield  the  appropriate  0  to  100  scaling  solution  for  the  SWAT  ratings,  the  prototype 
solutions  for  the  appropriate  subjects.  The  Kendall's  coefficient  of  concordance  (0.7256) 
for  the  individual's  card  sorts  was  less  than  the  recommended  .78  for  using  the  group 
scaling  solution,  so  prototype  solutions  were  used  instead.  Two  primary  dimensions 
were  found  from  the  prototype  analysis:  Stress  and  Time.  One  subject  was  found  to 
have  an  Effort  prototype,  but  was  switched  to  the  next  greatest  prototype  tendency, 
namely  Stress.  Of  the  16  subjects  in  the  analysis,  9  found  Stress  to  be  the  most 
important  dimension  in  the  card  sort,  while  the  remaining  7  found  Time  to  be  the  most 
important  dimension. 

SWAT  (In-Flight)  could  not  discriminate  among  the  three  workload  flights,  F  <  1, 
although  a  significant  workload  by  phase  of  flight  interaction  was  found,  F(1 2, 1 68) = 5. 1 1 , 
(MSe  =  1 92,  p  <  .01 )  (Figures  9.2.1 .1  -1  to  9.2. 1 .1  -4  and  Table  9.2. 1.1-1). 

A  comparison  of  the  Nominal-Malfunction  flights  found  a  significant  interaction  of 
workload  by  phase  of  flight,  F(6,84)=6.49,  (MSe  =  206,  pc. 01).  A  comparison  of  the 
Communication-Malfunction  flights  found  a  significant  interaction  of  workload  by  phase 
of  flight,  F(6,84) =7.32,  (MSe  =  191 ,  p  c  .01). 

A  main  effect  for  the  phase  of  flight  comparison  was  found  for  all  three  types  of  workload 
flights,  F(6,84) =25.59,  (MSe =344,  p<.01).  A  oneway  ANOVA  found  a  main  effect  for 
phase  of  flight  significant  for  the  Nominal  workload  flight,  F(6,90)  =  10.98,  (MSe =129, 
p<. 01).  A  Newman-Kuels  range  statistic  was  computed  to  determine  SWATs  ability  to 
discriminate  various  phases  of  flight  from  one  another  in  the  Nominal  workload  flight,  9 
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Figure  9.2. 1 . 1 -2 
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Figure  9.2. 1 . 1-3 

In-Flight  SWAT 
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Figure  9.2.1 .1-4 
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Table  9.2.1 .1 -1 


Subjective  Workload  Assessment  Technique 

(In-Flight  SWAT) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Nominal 

Communication 

Malfunction 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

24.8 

(19.8) 

24.5 

(18.9) 

20.0 

(16.1) 

Climb 

11.1 

(15.6) 

19.6 

(18.0) 

14.7 

(15.2) 

Top  of  climb 

9.1 

(12.8) 

11.7 

(13.5) 

1.4 

(3.9) 

Cruise 

2.9 

(5.5) 

8.6 

(16.2) 

14.1 

(13.0) 

Top  of  descent 

16.6 

(12.2) 

15.2 

(18.0) 

13.1 

(13.8) 

Approach 

15.9 

(16.6) 

19.3 

(20.1) 

32.2 

(14.2) 

Landing  or  M/A 

30.4 

(20.1) 

29.4 

(22.2) 

46.4 

(23.9) 

No.  1  engine  failure 

48.8 

(21.5) 

“A”  hydraulic  failure 

41.2 

(16.9) 

Approach 

36.0 

(18.9) 

Landing 

44.6 

(21.5) 

H 1167 .04  R2dD 


Table  9.2.1 .1-2 

SWAT  (In-Flight) 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight  1 

Communication  flight 

Malfunction  flight 

Takeoff 

0.50 

0.44 

0.68* 

Climb 

0.82* 

0.36 

0.74* 

Top  of  climb 

0.50 

0.39 

-0.07 

Cruise 

0.05 

0.72* 

0.40 

Top  of  descent 

0.52 

0.78* 

0.46 

Approach 

0.60 

0.76* 

0.48 

Landing  or 
missed  approach 

0.49 

0.58 

r(13)=  0.79* 

No.  1  engine  failure 

• 

o 

o 

ll 

CO 

“A"  system  hydraulics 

foili  irQ 

0.60 

IUIIUI  ^ 

Approach 

Landing 

r{  14)  =  0.623* 

*  Significant  p  <  0.01 

0.53 

r(13)=  0.63  ' 
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out  of  21  comparisons  were  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase 
of  flight  significant  for  the  Communication  workload  flight,  F(6,90)=9.89,  (MSe=84, 
P  c  .01 ).  The  ability  to  discriminate  phase  of  flight  conditions  for  the  Communication  flight 
found  6  out  of  21  comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for 
phase  of  flight  significant  for  the  Malfunction  workload  flight,  F(1 0,1 30) =22.47, 
(MSe=166,  p<. 01).  The  ability  to  discriminate  phase  of  flight  conditions  for  the 
Malfunction  flight  found  30  out  of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilots  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1. 1-2).  For  the  nominal  flight 
there  was  one  significant  correlation  out  of  a  possible  seven.  For  the  communication  and 
malfunction  flights  there  were  3  out  of  7,  and  3  out  of  11,  significant  correlations, 
respectively. 

In  assessing  inter-rater  reliability,  it  was  found  that  94%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

SWAT  fPost-Flioht) 

The  same  scaling  solution  was  used  for  determining  the  0-to-100  workload  for  the  In- 
Flight  event  ratings  was  used  for  the  Post-Flight  event  ratings.  The  same,  pre-flight,  card 
sort  determined  the  prototype  solutions  for  the  two  groups:  Time  and  Stress. 

SWAT  (Post-Flight)  found  a  main  effect  among  the  three  workload  flights,  F(2,28)  =  17.21 , 
(MSe=252,  pc.01).  In  addition,  a  significant  workload  by  phase  of  flight  interaction  was 
found  F(1 2, 168)  =  11. 52,  (MSe=206,  pc.01)  (Figures  9.2.1. 1-5  to  9.2.1. 1-8  and  Table 
9.2. 1.1 -3). 

A  comparison  of  the  Nominal-Malfunction  flights  found  a  significant  main  effect  of 
workload  and  an  interaction  of  work  load  by  phase  of  flight,  f  (1,1 4) =26.50,  (MSe=310, 
pc.01)  and  F(6,84)  =  13.32,  (MSe =271,  pc.01),  respectively.  A  comparison  of  the 
Communication-Malfunction  flights  found  a  significant  main  effect  of  workload  and  an 
interaction  of  workload  by  phase  of  flight,  F(1,14)  =  13.13,  (MSe=218,  pc.01)  and 
F(6,84)  =  1 2.68,  (MSe = 247,  p  c  .01 ),  respectively. 

A  main  effect  for  phase  of  flight  comparison  was  found  for  all  three  types  of  workload 
flights,  F(6,84)= 25.81,  (MSe=375,  pc.01).  A  oneway  ANOVA  found  a  main  effect  for 
phase  of  flight  significant  for  the  Nominal  workload  flight,  F(6,90)  =  11.52,  (MSe  =  129, 
pc.01).  A  Newman-Kuels  range  statistic  was  computed  to  determine  SWATs  ability  to 
discriminate  various  phases  of  flight  from  one  another  in  the  Nominal  workload  flight  7 
out  of  21  comparisons  were  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase 
of  flight  significant  for  the  Communication  workload  flight,  F(6,90)=7.24,  (MSe =121, 
p  c  .01 ).  The  ability  to  discriminate  phase  of  flight  conditions  for  the  Communication  flight 
found  7  out  of  21  comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for 
phase  of  flight  was  significant  for  the  Malfunction  workload  flight,  F(1 0,1 30) =28.32, 
(MSe  =  192,  pc.01).  The  ability  to  discriminate  phase  of  flight  conditions  for  the 
Malfunction  flight  found  33  out  of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1. 1-4).  For  the  nominal  flight 
there  were  two  significant  correlations  out  of  a  possible  seven.  For  the  communication 
and  malfunction  flights  there  were  3  out  7,  and  5  out  of  11,  significant  correlations, 
respectively. 
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Figure  9.2.1 .1-6 
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Figure  9.2. 1 .1-7 
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Figure  9.2.1 .1-8 
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Table  9.2.1 .1-3 


Subjective  Workload  Assessment  Technique 

(Post-Flight  SWAT) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window  Nominal 


Mean 

SD 

Takeoff 

25.6 

(20.2) 

Climb 

14.4 

(16.3) 

Top  of  climb 

3.9 

(7.7) 

Cruise 

1.3 

(13.1) 

Top  of  descent 

16.9 

(13.1) 

Approach 

14.9 

(14.2) 

Landing  or  M/A 

26.5 

(25.0) 

No.  1  engine  failure 
“A”  hydraulic  failure 
Approach 
Landing 


Communication 

Malfunction 

Mean 

SD 

Mean 

SD 

27.8 

(19.3) 

16.3 

(15.9) 

15.6 

(16.3) 

12.7 

(12.8) 

10.2 

(10.8) 

2.9 

(6.5) 

11.4 

(14.3) 

21.3 

(17.5) 

16.8 

(15.9) 

18.2 

(15.8) 

17.1 

(17.0) 

33.7 

(23.5) 

29.0 

(22.8) 

59.9 

(17.2) 

59.8 

(19.6) 

49.4 

(18.2) 

29.8 

(21.5) 

34.2 

(22.5) 

Ml  16?  05  R2db 


Table  9. 2. 1.1-4 

SWAT  (Post-Flight) 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight  Communication  flight 

Malfunction  flight 

Takeoff 

0.74* 

0.44 

0.42 

Climb 

0.59 

0.82* 

0.41 

Top  of  climb 

0.43 

-0.05 

0.71* 

Cruise 

-0.08 

0.31 

0.66* 

Top  of  descent 

0.15 

0.65* 

0.29 

Approach 

0.39 

0.60 

0.66* 

Landing  or 
missed  approach 

0.81* 

0.66* 

r(13)=  0.75* 

No.  1  engine  failure 

0.44 

“A”  system  hydraulics 

0.53 

laiiuio 

Approach 

Landing 

r(  14)  =  0.623* 

*  Significant  p  <  0.01 

0.55 
0.69*  " 
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In  assessing  inter-rater  reliability,  it  was  found  that  94%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 
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BEDFORD  RATING 

No  transformation,  from  event  rating  to  workload  score,  is  necessary  for  the  Bedford 
rating.  Pilots  made  ratings,  from  1-to-10,  low  to  high,  with  half  ratings  (i.e.,  3  1/2) 
allowed. 

Bedford  ratings  found  a  main  effect  for  workload,  F(2,28)  =  12.55,  (MSe  =  1,  p<.01).  In 
addition,  a  significant  workload  flight  by  phase  of  flight  interaction  was  found, 
F(12,168)  =  10.57,  (MSe=0.57, pc.01)  (Figures  9.2.1. 1-9  to  9.2.1.1-12  and  Table  9.2.1. 1- 
5). 

A  comparison  of  the  Nominal-Communication  flights  found  a  significant  main  effect  of 
workload  type,  F(1,15)=8.84,  (MSe=0.75,  pc.01).  A  comparison  of  the  Nominal- 
Malfunction  flights  found  a  main  effect  of  workload  and  an  interaction  of  workload  by 
phase  of  flight,  F(1,14)  =  17.96,  (MSe  =  1.4,  p<.01)  and  F(6,64)  =  18.60,  (MSe=0.5, 
p  <  .01 ).  A  comparison  of  the  Communication-Malfunction  flights  found  a  significant  main 
effect  of  workload  and  an  interaction  of  workload  by  phase  of  flight,  F(1,14)=8.71, 
(MSe=0.86,pc.01)  and  F(6, 84)  =  10.99,  (MSe=0.75,p<.01),  respectively. 

A  main  effect  for  phase  of  flight  discrimination  was  found,_F(6,84)  = 27.78,  (MSe=0.98, 
p<.0l).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Nominal  workload  flight,  F(6,90)  =  15.82,  (MSe=0.16,  pc.01).  A  Newman-Kuels  range 
statistic  was  computed  to  determine  the  ability  of  the  Bedford  ratings  to  discriminate 
various  phases  of  flight  from  one  another  in  the  Nominal  workload  flight,  1 1  out  of  21 
comparisons  were  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Communication  workload  flight,  F(6,90)=5.82,  (MSe=0.37, 
pc. 01).  The  ability  to  discriminate  phase  of  flight  conditions  for  the  Communication 
flight  found  6  out  of  21  comparisons  significant.  A  oneway  ANOVA  found  a  main  effect 
for  phase  of  flight  discrimination  for  the  Malfunction  workload  flight,  F(1 0,1 30) =25.58, 
(MSe=0.59,  pc.01).  The  ability  to  discriminate  phase  of  flight  conditions  for  the 
Malfunction  flight  found  35  out  of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1. 1-6).  For  the  Nominal  flight 
there  were  two  significant  correlation  out  of  a  possible  seven.  For  the  Communication 
and  Malfunction  flights  there  were  1  out  7,  and  1  out  of  11,  significant  correlations, 
respectively. 

In  assessing  inter-rater  reliability,  it  was  found  that  94%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

9.2.1. 2  PHYSIOLOGICAL  MEASURES 

Eyeblink  rate  and  heart  rate,  measured  by  inter-beat  interval,  both  demonstrate  evidence 
of  validity  by  discriminating  between  the  different  types  of  workload.  Eyeblinks,  heart 
rate,  and  the  blood  pressure  component  of  the  power  spectral  analysis  all  demonstrate 
evidence  for  test/retest  and  inter-rater  reliability. 

EYEBLINK 

Using  the  same  scoring  protocol  as  the  Part-Task  simulation,  a  strip  chart  printout  of  the 
vertical  electro-occulogram  analog  signal  was  analyzed  for  eyeblinks.  The  number  of 
eyeblinks  per  minute  was  then  computed  to  determine  eyeblink  rate. 
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Figure  9.2.1 .1-9 
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Figure  9.2.1.1-10 
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Figure  9.2  1.1-11 
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Figure  9.2.1.1-12 
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Table  9.2.1 .1-5 


Window 


Bedford  Ratings 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Nominal 


Communication 


Malfunction 


Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

3.0 

(0.85) 

3.3 

(1.18) 

2.7 

(0.76) 

Climb 

2.5 

(0.58) 

2.5 

(0.59) 

2.6 

(0.62) 

Top  of  climb 

2.1 

(0.56) 

2.4 

(0.80) 

1.8 

(0.59) 

Cruise 

1.9 

(0.55) 

2.4 

(0.99) 

3.0 

(0.85) 

Top  of  descent 

2.7 

(0.56) 

2.8 

(0.60) 

3.0 

(0.99) 

Approach 

2.7 

(0.62) 

2.9 

(0.72) 

3.6 

(0.83) 

Landing  or  M/A 

2.9 

(0.72) 

3.2 

(0.74) 

4.7 

(1.10) 

No.  1  engine  failure 

5.1 

(1.65) 

“A"  hydraulic  failure 

4.6 

(1.46) 

Approach 

3.6 

(1.26) 

Landing 

3.6 

(0.86) 

H  n6?0fc  R3cd 


Window 

Table  9.2.1 .1-6 

Bedford  Rating 

Full  Mission  Simulation 
Test-Retest 

Reliability  Correlations 

Nominal  flight  Communication  flight 

Malfunction  flight 

Takeoff 

0.45  0.64* 

0.61 

Climb 

0.64*  0.51 

0.31 

Top  of  climb 

0.40  0.01 

0.18 

Cruise 

0.26  0.15 

0.59 

Top  of  descent 

0.17  0.42 

0.55 

Approach 

0.58  0.44 

0.29 

Landing  or 

0.66*  0.04 

r(  13)  =  0.16 

missed  approach 

No.  1  engine  failure 

“A"  system  hydraulics 

fadure  r(14)  =  0.623* 

Approach  *  signjficant  p  <  0.01 

Landing  - 

0.63* 

0.50 

0.62 

r(  13)  =0.26  r"C7,*R8G 
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Blink  rate  found  a  strong  trend  for  a  main  effect  among  the  three  workload  flights, 
F(2,28)  =  5.59,  (MSe =33,  pc. 01 2).  In  addition,  a  workload  by  phase  of  flight  interaction 
was  found,  F(1 2, 120) =4.59,  (MSe=21,  pc.01)  (Figures  9.2.1. 2-1  to  9.2.1. 2-4  and  Table 
9. 2.1. 2-1). 

A  comparison  of  the  Nominal-Malfunction  flights  found  a  interaction  of  workload  by 
phase  of  flight,  F(6,60)=7.10,  (MSe= 19,  pc.01).  A  comparison  of  the  Communication- 
Malfunction  flights  found  a  main  effect  of  workload  and  an  interaction  of  workload  by 
phase  of  flight,  f (1,10)  =  10.63,  (MSe =34,  pc.01)  and  F(6, 60)  =  5.82,  (MSe  =  20,  pc.01), 
respectively. 

A  main  effect  for  phase  of  flight  discrimination  was  found.  F(6.84)=5.21.  (MSe=39, 
pc.01).  A  oneway  ANOVA  found  a  strong  trend  for  a  main  effect  for  phase  of  flight 
discrimination  for  the  Communication  workload  flight,  F(6,60)  =  2.44,  (MSe  =  11,  pc. 04). 
There  was  no  ability  to  discriminate  phase  of  flight  conditions  for  the  Communication 
flight.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Malfunction  workload  flight,  F(1 0,1 00) =6.36,  (MSe  =  22,  pc. 01).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  11  out  of  55 
comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2. 1.2-2).  For  the  Nominal  flight 
there  were  four  significant  correlations  out  of  a  possible  seven.  For  the  Communication 
and  Malfunction  flights  there  were  4  out  7,  and  6  out  of  11,  significant  correlations, 
respectively. 

In  assessing  inter-rater  reliability,  it  was  found  that  73%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

HEART  RATE  (INTER-BEAT  INTERVALS 

Heart  rate  was  calculated  using  the  same  method  as  the  Part-Task  simulation,  the  R-R 
interbeat  interval. 

Inter-beat  interval  showed  a  strong  trend  for  a  main  effect  for  discriminating  among  the 
three  workload  flights,  F(2,28)=2.98,  (MSe=9024,  pc. 07).  In  addition,  a  significant 
workload  by  phase  of  flight  interaction  was  found,  F(1 2, 168)  =4.03,  (MSe=465,  pc. 01) 
(Figures  9.2. 1 .2-5  to  9.2. 1 .2-8  and  Table  9.2. 1 .2-3). 

In  addition,  as  was  found  for  the  Part-Task  simulation,  a  strong  trend  for  a  main  effect  of 
session  (day  1  faster  than  day  2)  F(1,14)=8.30,  (MSe  =  56079,  pc.01).  The  slowing  of 
the  heart  rate  is  thought  to  reflect  a  "learning  effect"  from  the  test  to  the  retest  portion  of 
the  study.  A  comparison  of  the  Nominal-Communication  flights  found  a  main  effect  of 
session  (day  1  faster  than  day  2),  F(1, 14)  =  10.03,  (MSe =37251, pc.01).  Strong  trends 
for  session  main  effects  were  found  for  separate  comparisons  of  the  Nominal- 
Malfunction  and  Communication-Malfunction  flights,  F(1,14)=6.87,  (MSe =38558, 
P c. 02)  and  F(1, 14)  =7.88,  (MSe =37779, pc. 01),  respectively. 

A  comparison  of  the  Nominal-Malfunction  flights  found  a  strong  trend  for  a  main  effect  of 
workload  and  an  interaction  of  workload  by  phase  of  flight,  f(1,14)  =  5.83,  (MSe=6126, 
pc.03)  and  F(6,84)=4.38,  (MSe  =  527,  pc.01),  respectively.  A  comparison  of  the 
Communication-Malfunction  flights  found  a  strong  trend  for  a  main  effect  of  workload 
and  an  interaction  of  workload  by  phase' of  flight,  F(1,14)=4.16,  (MSe  =  10609,  pc.06) 
and  F(6,84)=6.34,  (MSe=394, pc.01),  respectively. 
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Figure  9.2. 1 .2-1 
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Figure  9. 2. 1.2-2 
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Figure  9.2.1 .2-3 
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Figure  9.2.1 .2-4 
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Table  9.2. 1 .2-1 

Eyeblink  Rate  (Blinks  per  Minute) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window  Nominal 


Mean 

SD 

Takeoff 

13.6 

(6.0) 

Climb 

12.2 

(10.2) 

Top  of  climb 

13.4 

(9.8) 

Cruise 

14.3 

(8.2) 

Top  of  descent 

11.8 

(7.9) 

Approach 

12.2 

(7.4) 

Landing  or  M/A 

13.4 

(5.8) 

No.  1  engine  failure 
“A”  hydraulic  failure 
Approach 
Landing 


Communication  Malfunction 


Mean 

SD 

Mean 

SD 

16.3 

(7.3) 

16.4 

(7.7) 

14.3 

(9.2) 

10.4 

(7.2) 

12.3 

(6.4) 

13.2 

(8.4) 

16.3 

(9.3) 

15.6 

(7.3) 

13.8 

(8.0) 

14.2 

(7.7) 

12.6 

(7.4) 

8.3 

(5.9) 

14.4 

(9.0) 

6.6 

(4  9) 

11.2 

(5.3) 

18.0 

(10.6) 

14.4 

(10.3) 

15.5 

(12.6) 

Hi  16?  07  R4iJb 

Table  9. 2. 1.2-2 

Eyeblink  Rate 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight  Communication  flight 

Malfunction  flight 

Takeoff 

0.74* 

0.82* 

0.78* 

Climb 

0.73 

0.62 

0.96* 

Top  of  climb 

0.91* 

0.71 

0.77* 

Cruise 

0.94* 

0.82* 

0.62 

Top  of  descent 

0.88* 

0.90* 

0.78* 

Approach 

0.01 

0.83* 

0.94* 

Landing  or 
missed  approach 

0.31 

0.59 

0.62 

No.  1  engine  failure 

0.52 

“A"  system  hydraulics 
failure 

0.77* 

Approach 

Landing 

r(9)  =  .735* 

*Signiiicant  p  .01 

0.41 

0.29 
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Figure  9.2.1 .2-5 
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Figure  9.2.1 .2-6 
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Figure  9.2.1 .2-7 
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Figure  9.2.1 .2-8 


Inter-Beat  Interval  (Msec) 

Full  Mission  Simulation 


760 

740 

720 

700 

Inter-beat  qqq 
interval 
(Msec)  660 

640 

620 


All  Flights  Mean  scores 


-  •  Nominal  flight 


600 

O' 


Takeoff  Climb  Top  of  Cruise  Top  of  Approach  Landing  No.  1 
climb  descent  or  engine 

(TOC)  (TOD)  missed  failure 

approach 

Phase  of  flight 


“A”  Approach  Landing 
system 
hydraulic 
failure 

HI  167  36  R30t> 


132 


Table  9.2.1 .2-3 


Window 


Inter-Beat  Interval  (Msec) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Nominal 


Communication 


Malfunction 


Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

714 

(150) 

706 

(152) 

701 

(136) 

Climb 

744 

(157) 

745 

(166) 

717 

(142) 

Top  of  climb 

732 

(148) 

746 

(162) 

735 

(137) 

Cruise 

744 

(147) 

739 

(154) 

729 

(138) 

Top  of  descent 

743 

(161) 

744 

(163) 

728 

(142) 

Approach 

731 

(157) 

735 

(163) 

700 

(146) 

Landing  or  M/A 

670 

(136) 

675 

(134) 

638 

(123) 

No.  1  engine  failure 

679 

(135) 

“A"  hydraulic  failure 

717 

(151) 

Approach 

700 

(143) 

Landing 

646 

(126) 

Hi  167  00  H4do 

Table  9.2.1 .2-4 

Inter-beat  Interval  (Msec) 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight 

Communication  flight 

Malfunction  flight 

Takeoff 

0.87 * 

0.89* 

0.84* 

Climb 

0.86* 

0.89* 

0.84* 

Top  of  climb 

0.88* 

0.89* 

0.82* 

Cruise 

0.86* 

0.87* 

0.83* 

Top  of  descent 

0.88* 

0.89* 

0.84* 

Approach 

0.89* 

0.89* 

0.83* 

Landing  or 
missed  approach 

0.89* 

0.90* 

0.89* 

No.  1  engine  failure 

0.87* 

“A”  system  hydraulics 
failure 

0.88* 

Annrnanh 

0.89* 

Landing  ‘Significant  p-  .01 

0.92* 
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A  main  effect  for  phase  of  flight  discrimination  was  found1_F(6,84) = 25.04,  (MSe=2889, 
pc  .01).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Nominal  workload  flight,  F(6,84)  =  17.09,  (MSe=633,  pc. 01).  A  Newman-Kuels  range 
statistic  was  computed  to  determine  SWATs  ability  to  discriminate  various  phases  of 
flight  from  one  another  in  the  Nominal  workload  flight,  6  out  of  21  comparisons  were 
significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for 
the  Communication  workload  flight,  F (6, 84) =2 1.36,  (MSe  =  504,  pc.01).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Communication  flight  found  1 1  out  of  21 
comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Malfunction  workload  flight,  F(10,140)  =  21.79,  (MSe  =  725,  pc.01). 
The  ability  to  discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  23  out 
of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9. 2. 1.2-4).  Seven,  out  of  seven 
possible,  correlations  were  found  to  be  significant  for  the  Nominal  and  Communication 
flights.  Eleven,  out  of  the  eleven  possible,  correlations  were  found  to  be  significant  for 
the  Malfunction  flight. 

In  assessing  inter-rater  reliability,  it  was  found  that  87%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

HEART  RATE  VARIABILITY  (IBI  STANDARD  DEVIATIONS 

Heart  rate  variability  is  the  standard  deviation  of  the  R-R  interbeat  interval  for  a  given 
measurement  period. 

The  standard  deviation  of  IBI  found  no  main  effect  among  the  three  workload  flights, 
F(2,28)  =  1 .10,  ns  (Figures  9.2.1 .2-9  to  9.2.1 .2-12  and  Table  9.2.1 .2-5). 

A  significant  session  by  phase  of  flight  interaction  was  found,  F(6,84)=8.64,  (MSe  =  159, 
pc.01),  indicating  instability  of  the  measure's  sensitivity  to  different  task  demands  over 
time.  A  comparison  of  the  Nominal-Communication  flights  found  an  interaction  of 
session  by  phase  of  flight,  F(6,84)=5.94,  (MSe  =  13Q,  pc.01).  A  comparison  of  the 
Nominal-Malfunction  flights  found  an  interaction  of  session  by  phase  of  flight, 
F(6,84)=7.74,  (MSe  =  143,  pc.01).  A  comparison  of  the  Communication-Malfunction 
flights  found  an  interaction  of  session  by  phase  of  flight,  F (6, 84)  =6.81,  (MSe=140, 
pc.01),  respectively. 

A  main  effect  for  phase  of  flight  discrimination  was  found,_F(6,84)=3.76,  (MSe  =  274, 
pc.oi).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  disrhmination  for  the 
Communication  workload  flight,  F(6,84)  =  3.92,  (MSe=82,  pc. 04).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Communication  flight  found  1  out  of  21 
comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Malfunction  workload  flight,  F(10,140)=3.45,  (MSe  =  114,  pc.01). 
The  ability  to  discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  3  out 
of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (T able  9.2.1 .2-6).  No  significant  correlation 
were  found  for  the  Nominal  or  Communication  flights.  For  the  Malfunction  flight  there 
was  1  significant  correlation  out  of  a  possible  11." 

In  assessing  inter-rater  reliability,  it  was  found  that  33%  of  the  subjects  scores  were 
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Figure  9.2.1 .2-10 
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Figure  9.2.1.2-11 
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Figure  9.2.1.2-12 
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Table  9. 2. 1.2-5 

Inter-Beat  Interval 
Standard  Deviation  (Msec) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Nominal 

Communication 

Malfunction 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

41.93 

(14.73) 

40.27 

(18.07) 

39.03 

(13.91) 

Climb 

47.87 

(17.52) 

44.70 

(15.52) 

46.93 

(19.50) 

Top  of  climb 

48.87 

(15.77) 

48.60 

(15.13) 

45.40 

(13.80) 

Cruise 

50.70 

(18.41) 

51.90 

(18.28) 

49.47 

(20.32) 

Top  of  descent 

45.73 

(12.42) 

49.03 

(14.58) 

45.27 

(16.63) 

Approach 

51.53 

(17.99) 

46.90 

(16.64) 

44.47 

(15.50) 

Landing  or  M/A 

50.63 

(17.18) 

54.33 

(21.94) 

45.83 

(13.07) 

No.  1  engine  failure 

53.53 

(15.79) 

"A"  hydraulic  failure 

54.13 

(23.71) 

Approach 

49.33 

(15.45) 

Landing 

56.80 

(16.65) 

Table  9. 2. 1.2-6 

Inter-Beat  Interval 
Standard  Deviation  (Msec) 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight 

Communication  flight 

Malfunction  flight 

Takeoff 

0.01 

0.46 

0.51 

Climb 

-0.04 

0.32 

0.31 

Top  of  climb 

0.21 

0.28 

0.21 

Cruise 

0.15 

0.48 

0.69* 

Top  of  descent 

-0.07 

0.27 

0.62 

Approach 

0.25 

0.17 

0.51 

Landing  or 

0.17 

0.38 

0.15 

missecf  approach 

* 

No.  1  engine  failure 

0.35 

“A”  system  hydraulics 

0.60 

failure 

Approach 

0.58 

Landing  r(i3) 

=  .641* 

-0.09 

‘Significant  p-  .01 
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significantly  correlated  with  means  for  the  25  measurement  windows. 

POWER  SPECTRAL  ANALYSIS 

Power  spectral  analyses  were  computed  for  the  blood  pressure  (0.05  to  0.15  Hz)  and 
respiration  (0.20  to  0.40  Hz)  components  using  a  fast  fourier  transform  of  the  inter-beat 
interval  information. 

BLOOD  PRESSURE  COMPONENT 

The  blood  pressure  component  found  no  main  effect  among  the  three  workload  flights,  F 
<  1  (Figures  9.2.1.2-13  to  9.2.1.2-16  and  Table  9.2.1. 2-7). 

A  strong  trend  for  a  three-way  interaction  was  found  for  session  by  workload  by  phase  of 
flight,  F(12,168)  =  2.13,  (MSe=0.58,  pc. 02).  The  interaction  was  not  predicted,  and 
indicates  the  instability  of  the  measure  over  testing  periods.  When  comparing  the 
Nominal-Communication  another  strong  trend  for  a  three-way  interaction  was  found  for 
session  by  workload  by  phase  of  flight,  f  (6, 84)  =  1.95,  (MSe=0.64,  pc.08).  In  addition, 
comparing  the  Communication-Malfunction  another  strong  trend  for  a  three-way 
interaction  was  found  for  session  by  workload  by  phase  of  flight,  F(6,84)=2.70, 
(MSe=0.61,pc.02). 

A  main  effect  for  phase  of  flight  discrimination  was  found,_F(6,84)  =  12.64,  (MSe=1.68, 
pc. 01).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Nominal  workload  flight,  F(6,84)  =  5.12,  (MSe=0.80,  pc.01).  The  ability  to  discriminate 
phase  of  flight  conditions  for  the  Nominal  flight  found  2  out  of  21  comparisons  significant. 
A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Communication  workload  flight,  F(6,84)=8.06,  (MSe=0.58,  pc.01).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Communication  flight  found  6  out  of  21 
comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Malfunction  workload  flight,  F(10, 140)  =8.14,  (MSe=0.51,  pc.01). 
The  ability  to  discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  13  out 
of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1 .2-8).  No  significant  correlations 
were  found  for  the  Nominal  flight,  ^or  the  Communication  and  Malfunction  flights  there 
were  2,  out  of  a  possible  7,  and  3,  out  of  a  possible  1 1 ,  significant  correlations. 

In  assessing  inter-rater  reliability,  it  was  found  that  80%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

RESPIRATION  COMPONENT 

The  respiration  component  found  no  main  effect  among  the  three  workload  flights,  F  c  1 
(Figures  9.2.1.2-17  to  9.2.1.2-20  and  Table  9.2.1. 2-9). 

A  main  effect  for  phase  of  flight  discrimination  was  foundt_F(6,84)=6.81,  (MSe=0.07, 
pc.01).  A  oneway  ANOVA  found  a  strong  trend  for  a  main  effect  for  phase  of  flight 
discrimination  for  the  Nominal  workload  flight,  F(6,84)=2.51,  (MSe=0.036,  pc.03).  A 
oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Communication  workload  flight, f (6, 84) =3.03,  (MSe= 0.06,  pc.01).  A  oneway  ANOVA 
found  a  main  effect  for  phase  of  flight  discrimination  for  the  Malfunction  workload  flight, 
F(1 0,1 40)  =  2.99,  (MSe=0.01,  pc.01).  Although  the  separate  oneway  ANOVAs  found  a 
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Figure  9.2.1 .2-13 
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Figure  9.2.1 .2-14 
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Figure  9.2.1.215 
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Figure  9.2.1.2-16 
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Table  9.2.1 .2-7 

Power  Spectral  Analysis  (Blood  Pressure  Component) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 


Nominal 


Communication 


Malfunction 


Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

6.2 

(5.1) 

7.5 

(12.3) 

5.5 

(7.3) 

Climb 

9.6 

(10.9) 

7.3 

(5.9) 

10.0 

(9.5) 

Top  of  climb 

16.7 

(20.6) 

13.0 

(7.5) 

11.6 

(8.1) 

Cruise 

15.8 

(15.3) 

20.5 

(17.6) 

14.3 

(13.1) 

Top  of  descent 

10.0 

(6.2) 

13.9 

(9.7) 

14.4 

(11.8) 

Approach 

11.6 

(7.4) 

10.6 

(12.2) 

9.5 

(6.1) 

Landing  or  M/A 

1.8 

(1.8) 

3.4 

(4.7) 

1.8 

(1.2) 

No.  1  engine  failure 

11.8 

(8.8) 

“A"  hydraulic  failure 

18.8 

(18.6) 

Approach 

9.5 

(7.3) 

Landing 

1.6 

(1.7) 

H1 167  10  R4<to 

Table  9. 2. 1.2-8 

Power  Spectral  Analysis 
(Blood  Pressure  Component) 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight 

Communication  flight 

Malfunction  flight 

Takeoff 

0.54 

0.93* 

0.96* 

Climb 

0.28 

0.18 

0.61 

Top  of  climb 

0.59 

0.16 

0.44 

Cruise 

0.51 

0.38 

0.86* 

Top  of  descent 

-0.18 

0.59 

0.48 

Approach 

0.17 

0.66* 

0.21 

Landing  or 

0.32 

0.50 

0.19 

missed  approach 

No.  1  engine  failure 

0.53 

“A"  system  hydraulics 

0.69* 

failure 

Approach 

0.43 

Landing  r(i3) 

-  .641* 

0.48 

|  ’Significant  p-  .01  | 
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Figure  9.2.1.2-17 
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Figure  9.2.1 .2-18 
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Figure  9.2.1 .2-19 
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Figure  9.2.1 .2-20 
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Table  9.2.1 .2-9 

Power  Spectral  Analysis  (Respiration  Component) 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 


Nominal 


Communication 


Malfunction 


Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

2.4 

(2.7) 

2.1 

(2.2) 

1.5 

(0.9) 

Climb 

3.2 

(3.9) 

2.0 

(2.1) 

2.4 

(2.1) 

Top  of  climb 

3.0 

(2.3) 

3.5 

(3.9) 

2.2 

(2.3) 

Cruise 

2.7 

(1.8) 

3.0 

(2.3) 

2.5 

(2.2) 

Top  of  descent 

3.7 

(5.7) 

3.6 

(3.6) 

2.2 

(2.2) 

Approach 

3.6 

(3.0) 

4.3 

(7.7) 

2.6 

(1.9) 

Landing  or  M/A 

1.4 

(2.6) 

1.3 

(15) 

1.0 

(0.8) 

No.  1  engine  failure 

1.5 

(0.9) 

“A”  hydraulic  failure 

2.4 

(2.1) 

Approach 

2.2 

(2.3) 

Landing 

2.5 

(2.2) 

hi  >67  11  R4dO 

Table  9.2.1.2-10 

Power  Spectral  Analysis 
(Respiration  Component) 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 


Window 

Nominal  flight 

Communication  flight 

Malfunction  flight 

Takeoff 

-0.14 

-0.14 

0.16 

Climb 

-0.10 

-0.15 

-0.04 

Top  of  climb 

-0.17 

-0.13 

0.13 

Cruise 

-0.25 

-0.22 

0.16 

Top  of  descent 

-0.13 

-0.16 

0.40 

Approach 

-0.24 

-0.15 

0.21 

Landing  or 

-0.28 

0.13 

-0.26 

missed  approach 

No.  1  engine  failure 

-0.39 

"A"  system  hydraulics 

0.11 

failure 

Approach 

0.13 

Landing  I  r(i3) 

-  .641* 

-0.17 

|  'Significant  p  01 
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significant  main  effect  for  phase  of  flight  discrimination,  the  Newman-Kuels  range  statistic 
could  not  discriminate  any  of  the  phases  of  flight  from  one  another  within  the  individual 
flights. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1.2-10).  No  significant 
correlations  were  found  for  the  Nominal,  Communication,  or  Malfunction  flight. 

In  assessing  inter-rater  reliability,  it  was  found  that  47%  of  the  subjects  scores  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

9.2.1. 3  PERFORMANCE  MEASURES 

Control  input  activity  was  measured  by  calculating  flight  control  position  changes  over 
time.  The  position  of  the  flight  controls  was  collected  at  a  rate  of  10  Hz.  If  the  position  of 
a  flight  control  (i.e.,  wheel,  column,  or  pedals)  moved  more  than  2.5%  of  the  total  throw 
4  available  then  the  input  activity  counter  was  incriminated.  Each  control  input  activity 

index  is  computed  over  time,  so  that  control  activity  is  expressed  in  inputs  per  minute. 
Following  normal  convention,  wheel  input  activity  controls  roll,  column  input  activity 
controls  pitch,  and  pedal  input  activity  controls  yaw. 

*  WHEEL  (AILERON)  CONTROL  INPUTS 

Wheel  control  input  activity  found  a  main  effect  among  the  three  workload  flights, 
F(2,30)  =  13.91,  (MSe=52,  pc. 01).  A  workload  by  phase  of  flight  interaction  was  found 
as  well,  F(12, 180)  =9.74,  (MSe=35,  pc.01)  (Figures  9.2.1.3-1  to  9.2.1.3-4  and  Table 
9.2.1. 3-1). 

In  addition,  a  three  way  interaction  of  session  by  workload  by  phase  of  flight  was  found 
to  be  significant,  F(12,180)=2.29,  (M$e=25,  pc.01).  This  interaction  shows  some 
instability  of  the  measure  in  assessing  workload  differences  with  repeated  testing. 
Another  three  way  interaction  of  the  same  factors  was  found  when  comparing  the 
Nominal-Malfunction  flights,  f  (6, 90)  =4.54,  (MSe  =  22, p<.01). 

A  comparison  of  the  Nominal-Communication  flights  found  a  main  effect  of  workload  and 
an  interaction  of  workload  by  phase  of  flight,  F(1,15)=8.84,  (MSe=0.75,  pc.01)  and 
F(6,90)  =  13.33,  (MSe=32,  pc.01),  respectively.  A  comparison  of  the  Nominal- 
Malfunction  flights  found  a  significant  main  effect  of  workload  and  an  interaction  of 
workload  by  phase  of  flight,  F(1,15)  =  22.26,  (MSe=57,  pc.01)  and  F(6,90)  =  18.60, 
(MSe  =  10.90,  pc.01),  respectively.  A  comparison  of  the  Communication-Malfunction 
<  flights  found  an  interaction  of  workload  by  phase  of  flight,  F(6,90)=3.97,  (MSe=29, 

pc.01),  respectively. 

A  main  effect  for  phase  of  flight  discrimination  was  found,_F(6, 90) =84.50,  (MSe=80, 
pc .01).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 

•  Nominal  workload  flight,  F(6,90)= 56.81,  (MSe=37,  pc.01).  A  Newman-Kuels  range 
statistic  was  computed  to  determine  SWATs  ability  to  discriminate  various  phases  of 
flight  from  one  another  in  the  Nominal  workload  flight,  10  out  of  21  comparisons  were 
significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for 
the  Communication  workload  flight,  F(6, 90) =35.93,  (MSe=20,  pc.01).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Communication  flight  found  8  out  of  21 
comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Malfunction  workload  flight,  F(1 0,1 50)  =  71. 69,  (MSe=37,  pc.01). 
The  ability  to  discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  35  out 
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rigure  9.2.1 .3-1 
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Takeoft  Climb  Top  ol  Cruise  Top  of  Approach  Landing 
climb  descent 


Phase  of  flight 


Figure  9. 2. 1.3-2 

Wheel  (Aileron)  Control  Inputs  (per  Minute) 
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Figure  9.2. 1 .3-3 
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Figure  9.2.1 .3-4 
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Takeoff  Climb  Top  of  Cruise  Top  of  Approach  Landing  No.  1  -A"  Approach  Landing 

climb  descent  or  engine  system 

(TOC)  (TOD)  missed  lailure  hydraulic 

approach  failure 

Phase  of  flight 


147 


Table  9.2. 1 . 3-1 

Wheel  (Aileron)  Control  Input 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 


Nominal 


Communication 


Malfunction 


Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

9.4 

(4-7) 

69 

(4.4) 

4.4 

(2.9) 

Climb 

12  2 

(7.5) 

6.9 

(5.1) 

9.8 

(5.4) 

Top  of  climb 

7.4 

(4.3) 

7.6 

(5.8) 

2.6 

(2.2) 

Cruise 

2.5 

(2.7) 

25 

(2.4) 

1.3 

(15) 

Top  of  descent 

3.8 

(3.5) 

4.8 

(3.4) 

5.0 

(2.6) 

Approach 

10.8 

(4.8) 

11.2 

(6.3) 

13.1 

(7.3) 

Landing  or  M/A 

36  4 

(15.6) 

23.0 

(10.6) 

22  6 

(9.3) 

No.  1  engine  failure 

18.4 

(8.7) 

"A"  hydraulic  failure 

8.2 

(5.2) 

Approach 

17.9 

(6.7) 

Landing 

46.0 

(16.5) 

Table  9.2. 1 .3-2 

Wheel  (Aileron)  Control  Inputs 

Full  Mission  Simulation 

Test-Retest 

Reliability  Correlations 

Window 

Nominal  flight  Communication  flight 

Malfunction  flight 

Takeoff 

0.38 

0.11 

0.36 

Climb 

0.70* 

0.69* 

0.59 

Top  of  climb 

0.72* 

0.73* 

0.64* 

Cruise 

-0.49 

0.46 

0.05 

Top  of  descent 

0.40 

0.36 

0.30 

Approach 

0.78* 

0.58 

0.66* 

Landing  or 
missed  approach 

0  51 

0.30 

0.51 

No.  1  engine  failure 

0.75* 

"A”  system  hydraulics 

0.72* 

r(  14)  0.623* 

Approach  *  Significant  p 

0.01 

0.47 

LcU  iuii  ly 

0.72*  . . . 
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of  55  comparisons  significant. 


Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1. 3-2).  For  the  Nominal  flight 
there  was  three  significant  correlation  out  of  a  possible  seven.  For  the  Communication 
and  Malfunction  flights  there  were  2  out  7,  and  5  out  of  11,  significant  correlations, 
respectively. 

In  assessing  inter-rater  reliability,  it  was  found  that  all  of  the  subjects  scores  (100%)  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

COLUMN  (ELEVATOR)  CONTROL  INPUTS 

Column  control  input  activity  found  a  main  effect  among  the  three  workload  flights, 
F(2,30)=20.17,  (MSe  =  27,  pc. 01).  A  workload  by  phase  of  flight  interaction  was  found 
as  well,  f(1 2, 180)  =  18.88,  (MSe  =  27,  pc.01)  (Figures  9.2.1.3-5  to  9.2.1.3-8  and  Table 
9.2.1. 3-3). 

A  comparison  of  the  Nominal-Communication  flights  found  a  main  effect  of  workload  and 
an  interaction  of  workload  by  phase  of  flight,  F(1, 15) =27.60,  (MSe=24,  pc.01)  and 
F(6, 90) =23.27,  (MSe  =  26,  p < .01),  respectively.  A  comparison  of  the  Nominal- 
Malfunction  flights  found  a  significant  main  effect  of  workload  and  an  interaction  of 
workload  by  phase  of  flight,  F(1, 15)  =33.39,  (MSe=29,  pc.01)  and  F(6, 90)  =28.04, 
(MSe=31,pc.oi),  respectively.  A  comparison  of  the  Communication-Malfunction  flights 
found  a  strong  trend  for  an  interaction  of  workload  by  phase  of  flight,  F(6,90)=2.58, 
(MSe=24, pc.01). 

A  main  effect  for  phase  of  flight  discrimination  was  foundJ:(6,90)  =  169.83,  (MSe  =  103, 
pc  .01).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Nominal  workload  flight,  F(6,90)= 209.36,  (MSe  =  24,  pc.01).  A  Newman-Kuels  range 
statistic  was  computed  to  determine  SWATs  ability  to  discriminate  various  phases  of 
flight  from  one  another  in  the  Nominal  workload  flight,  10  out  of  21  comparisons  were 
significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for 
the  Communication  workload  flight,  F(6, 90)  =99.34,  (MSe=22,  pc.01).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Communication  flight  found  10  out  of  21 
comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Malfunction  workload  flight,  F(1 0,1 50)  =  79.92,  (MSe=45,  pc.01). 
The  ability  to  discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  22  out 
of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot’s  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9. 2. 1.3-4).  For  the  Nominal  and 
Communication  flights  none  of  the  measurement  windows  showed  a  significant 
correlation.  For  the  Malfunction  flights  there  were  3,  out  of  11  possible,  significant 
correlations. 

In  assessing  inter-rater  reliability,  it  was  found  that  all  of  the  subjects  scores  (100%)  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

PEDAL  (RUDDER)  CONTROL  INPUTS 

Pedal  control  input  activity  found  a  main  effect  among  the  three  workload  flights, 
F(2,30)  =  26.43,  (MSe  =  15,  pc.01).  A  workload  by  phase  of  flight  interaction  was  found 
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Figure  9.2.1 .3-5 
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Figure  9.2.1 .3-6 
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Figure  9.2. 1 .3-7 
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Figure  9.2.1 .3-8 
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Table  9.2.1 .3-3 


Column  (Elevator)  Control  Input 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 


Nomina! 


Communication 


Malfunction 


Mean 

SD 

Mean 

SD 

Mean 

SD 

Takeoff 

8.1 

(4.4) 

7.9 

(3.5) 

6.7 

(2.9) 

Climb 

2.3 

(2.7) 

2.0 

(1.7) 

1.9 

(2.1) 

Top  of  climb 

3.5 

(3.0) 

2.4 

(2.0) 

0.8 

(1.0) 

Cruise 

0.7 

(1.0) 

1.2 

(1.6) 

0.3 

(0.4) 

Top  of  descent 

0.8 

(0.6) 

1.0 

(1.1) 

0.3 

(0.4) 

Approach 

4.7 

(3.8) 

4.8 

(5.4) 

8.4 

(6.6) 

Landing  or  M/A 

50.0 

(12.1) 

33.6 

(11.3) 

31.0 

(14.1) 

No.  1  engine  failure 

10.0 

(5.3) 

“A”  hydraulic  failure 

3.0 

(2.0) 

Approach 

8.1 

(6.3) 

Landing 

47.7 

(17.4) 

Hi  16?  13  R3co 

Table  9. 2. 1.3-4 

Column  (Elevator)  Control  Inputs 

Full  Mission  Simulation 
Test- Retest 
Reliability  Correlations 


Window 

Nominal  flight 

Communication  flight 

Malfunction 

Takeoff 

0.54 

0.53 

0.41 

Climb 

0.39 

0.28 

0.75* 

Top  of  climb 

0.33 

0.03 

0.32 

Cruise 

-0.19 

0.03 

-0.31 

Top  of  descent 

-0.11 

0.17 

0.66* 

Approach 

0.28 

0.52 

0.72* 

Landing  or 

0.01 

-0.52 

0.43 

missed  approach 

No.  1  engine  failure 

0.55 

“A"  system  hydraulics 

0.38 

,allure  frfW 

-  0.623* 

1 

Approach  *  significant  p  <  0.01 

Landing  - 

J 

0.44 

0.31 

152 


as  well,  f(1 2, 180)  =  12.59,  (MSe  =  17,  p<.01)  (Figures  9.2. 1.3-9  to  9.2.1.3-12  and  Table 

9.2. 1.3- 5). 

A  comparison  of  the  Nominal-Communication  flights  found  a  strong  trends  for  both  a 
main  effect  of  workload  and  an  interaction  of  workload  by  phase  of  flight,  F(1,15)=7.26, 
(MSe  =  12, p<. 01)  and  F(6, 90)  =  2.84,  (MSe  =  17, p<.01),  respectively.  A  comparison  of 
the  Nominal-Malfunction  flights  found  a  significant  main  effect  of  workload  and  an 
interaction  of  workload  by  phase  of  flight,  F(1,15)=41.13,  (MSe  =  19,  p<.01)  and 
F(6,90)  =  16.46,  (MSe=22,  p<.01),  respectively.  A  comparison  of  the  Communication- 
Malfunction  flights  found  a  for  main  effect  of  workload  and  an  interaction  of  workload  by 
phase  of  flight,  ,  f (1,1 5) =23.70,  (MSe  =  14,  p<.01)  and  f (6, 90)  =  19.07,  (MSe  =  12, 
P<.01),  respectively. 

A  main  effect  for  phase  of  flight  discrimination  was  found  JF(6, 90)  =37.96,  (MSe  =  106, 
p<  .01).  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for  the 
Nominal  workload  flight,  F(6, 90)  =30.63,  (MSe=36,  pc.01).  A  Newman-Kuels  range 
statistic  was  computed  to  determine  SWATs  ability  to  discriminate  various  phases  of 
flight  from  one  another  in  the  Nominal  workload  flight,  1 1  out  of  21  comparisons  were 
significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight  discrimination  for 
the  Communication  workload  flight,  F(6, 90)  =32.19,  (MSe=22,  pc.01).  The  ability  to 
discriminate  phase  of  flight  conditions  for  the  Communication  flight  found  10  out  of  21 
comparisons  significant.  A  oneway  ANOVA  found  a  main  effect  for  phase  of  flight 
discrimination  for  the  Malfunction  workload  flight,  F(1 0,1 50)  =  13.93,  (MSe=27,  pc.01). 
The  ability  to  discriminate  phase  of  flight  conditions  for  the  Malfunction  flight  found  18  out 
of  55  comparisons  significant. 

Test-retest  reliability  was  computed  by  comparing  the  pilot's  ratings  for  session  one  and 
two  for  the  separate  measurement  windows  (Table  9.2.1. 3-6).  For  the  Nominal  there 
were  two  significant  correlations  out  of  seven.  For  the  Communication  flight  none  of  the 
measurement  windows  showed  a  significant  correlation.  For  the  Malfunction  flights  there 
were  3,  out  of  1 1  possible,  significant  correlations. 

In  assessing  inter-rater  reliability,  it  was  found  that  all  of  the  subjects  scores  (100%)  were 
significantly  correlated  with  means  for  the  25  measurement  windows. 

9.3  RELATIONSHIP  OF  MEASURES 

As  was  done  for  the  Part-Task  simulation,  a  correlation  matrix  was  generated  for  the 
means  for  each  of  the  workload  measures  from  the  25  measurement  windows  (Table 

9.3- 1). 


Again  the  reader  should  use  caution  when  drawing  conclusions  from  the  correlation 
matrix.  Similar  to  the  Part-Task  simulation,  the  correlation  matrix  is  computed  on  the 
means  for  the  various  measures  from  the  various  workload  windows:  seven  windows 
each  from  the  Nominal  and  Communication  flights,  along  with  the  11  from  the 
Malfunction  flight.  The  correlation  matrix  should  be  viewed  simply  as  an  aid  in 
understanding  the  global  trends  of  covariation  of  the  workload  measures  in  assessing 
the  task  demands  of  the  three  flights. 

9.4  PRINCIPAL  COMPONENT  ANALYSIS 

Another  Principal  Component  Analysis  (PCA)  was  performed  on  the  data  from  the  Full- 
Mission  simulation  (Table  9.4-1).  The  interpretation  of  the  factor  loadings  in  any  sort  of  a 
factor  analysis  should  be  done  with  caution. 


Figure  9.2.1 .3-9 

Pedal  (Rudder)  Control  Inputs  (per  Minute) 

Full  Mission  Simulation 
Nominal  Flight  SFO  -  SCK 


Pedal 

control 

inputs/ 

min 


•  Mean  scores 

T  ±  1  standard 
-L  deviation 


Takeoff  Climb  Top  of  Cruise  Top  of  Approach  Landing 
climb  descent 

(TOC)  (TOD) 

Phase  of  flight 


Figure  9.2.1 .3-10 

Pedal  (Rudder)  Control  Inputs  (per  Minute) 

Full  Mission  Simulation 
Communications  Flight  SMF  -  SFO 


Pedal 

control 

inputs/ 

min 


A  Mean  scores 

T  ±  1  standard 
I  deviation 


Takeoff  Climb  Top  of  Cruise  Top  of  Approach  Landing 
climb  descent 

(TOC)  (TOD) 

Phase  of  flight 
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Figure  9.2.1 .3-11 

Pedal  (Rudder)  Control  Inputs  (per  Minute) 

Full  Mission  Simulation 
Malfunction  Flight  LAX  -  (SFO)  -  (OAK)  -  SMF 


Pedal 

control 

inputs/ 


X  Mean  scores 

T  ±  1  standard 
-L  deviation 


Takeoff 


Top  of  Cruise  Top  of  Approach  Missed  No.  1  "A"  Approach  Landing 

climb  descent  approach  engine  system 

(TOC)  (TOD)  failure  hydraulic 

Phase  of  flight  lailure 

ci  ic?  -»c  mr 


Figure  9.2.1.3-12 

Pedal  (Rudder)  Control  Inputs  (per  Minute) 

Full  Mission  Simulation 


Peda! 

control 

inputs/  20 
min 


All  Flights 


Mean  scores 
•  Nominal  flight 
A  Communication  flight 
X  Malfunction  flight 


Takeoff 


Top  of  Cruise  Top  of  Approach  Landing  No.  1  “A"  Approach  Landing 

climb  descent  or  engine  system 

(TOC)  (TOD)  missed  failure  hydraulic 

approach  failure 

Phase  of  flight 
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Table  9.2.1 .3-5 


Pedal  (Rudder)  Control  Input 

Full  Mission  Simulation  Data 
Means  and  Standard  Deviations 


Window 

Nominal 

Communication 

Malfunction 

Mean 

SO 

Mean 

SD 

Mean 

SD 

Takeoff 

19.70 

(10.2) 

15.20 

(8.1) 

13.50 

(8.1) 

Climb 

0.10 

(0.3) 

0.00 

(0.0) 

0.00 

(0.0) 

Top  of  climb 

0.03 

(0.1) 

0.00 

(0.0) 

0.00 

(0.0) 

Cruise 

0.00 

(0.0) 

0.00 

(0.0) 

0.00 

(00) 

Top  of  descent 

0.00 

(0.0) 

0.03 

(0.1) 

0.03 

(0.1) 

Approach 

0.10 

(0.4) 

0.20 

(0.2) 

0.10 

(04) 

Landing  or  M/A 

13.70 

(12.9) 

11.90 

(10.2) 

1.60 

(5.0) 

No.  1  engine  failure 

2.20 

(2.6) 

"A"  hydraulic  failure 

0.05 

(0.1) 

Approach 

0.80 

(1.7) 

Landing 

11.30 

(16.0) 

HI  167  14  HZC'o 

Window 


Table  9. 2. 1.3-6 

Pedal  (Rudder)  Control  Inputs 

Full  Mission  Simulation 
Test-Retest 
Reliability  Correlations 

Nominal  flight  Communication  flight  Malfunction  flight 


Takeoff 

Climb 

Top  of  climb 
Cruise 

Top  of  descent 
Approach 
Landing  or 
missed  approach 
No.  1  engine  failure 

“A"  system  hydraulics 
farfure 

Approach  *  Sj( 

Landing  - 


0.57 

0 

0 

0 

0 

0.73* 

0.66* 


r(  14)  =  0.623* 

*  Significanl  p  <  0.01 


0.59 

0 

0 

0 

0 

1.0* 

0.77* 

0.84* 

•0.10 

0.57 

0.77* 


56 


Correlation  Matrix 

Full  Mission  Simulation 


Table  9.3-1 


0(0  0) 
O  (O  CM 

"■'do 


O  00  CM  LO 

O  -r-  CM  ■<— 

I-1  O  O  O 


O  CM  o  O)  CO 
O  CM  fv.  LO  CM 

-  o  6  o  o 


o  o  coij  if  O) 
O  in  CM  to  lO  CM 


99° 


oioiosmoo 
p  in  co  co  rv  to  ^ 

r'd^°oo9 


OP)t-CVN0)’-(0 

-d°d°69? 

■  i  1  • 


O  f~  CO  O  LO  CM  LO  O  CO 
O  LO  CM  fs.  CM  CO  t—  CM 

^odd^dooci 


OVO’-CMWWO^N 

O0)(0(0«00C\Jt0r-  r- 

-ddd°doooci 


OCMOO'^COCMOOIO'^CMO 
O0»0)^0r-  <0  CM  CM  O  CM 

^ododd^dodo 


u_  |_  tr  .  _  Q 

0^5=:^  WCL(/)^ 

uj^wih-Qccccmirm 

coS£q_^coq_xi<<uj 


Q>  2 

^.E  0) 

c  EB 

c  a>  (z 
»_  Q.  C- 
a>  v_ 
Q.^  0) 

o  a 


CD 

CO  — 

cn  co 

(fl  C  ' — " 

ar-=  -s 

ro  c  2  o 

o  vss  ci. 

W  «  I—  .£ 

.E^il 

e^€,_§ 

or  =  a 

a>  ==  o  > 

CD  £  cl  > 


ca 

>Q)  _ 

—  T3  £0 

O^TD  cE 

Its 

—  c 

2  _  -0 
c  o  a> 

8|2 

gSg, 

t  7=  CO 

n  lu  k. 

aj 

O  CD  > 
O  CL  < 


CO  •" 
CD  C- 
-O  0) 
k—  c 
CD  O 

-  E 

CD  o 
CD  O 
CD 

aj  ol 

<m 

i-  £2 

O  ^ 

c  lo 
o  c 

I< 

>  co 
Q  o 

E# 

I  <0 
“O  m 

si 

CO  CL 


O 

-9  LL  t_  t  i  _  Q 

•c  0>5r^-J i2wq. 
co  uj>coiHQQCcm 
>  CQ^Q.5«0.XI< 


157 


ARS  Power  Spectral  Analysis  Respiration  component 

EBK  Eye  blinks  per  minute 

Critical  correlation  values  are  r{23)  =  0.505  or  r(23)  =  -0.505  are  in  bold 


Table  9.4-1 


Principal  Component  Analysis 

Full  Mission  Simulation 


Sorted  rotated  factor  loadings  (pattern) 


Factor  1 

Factor  2 

Factor  3 

Factor  4 

Blood  pressure 

0.893 

Respiration 

0.867 

Column  control  input 

-0.814 

-0.306 

0.461 

Pedal  control  input 

-0.742 

0.502 

Average  IBI 

0.690 

-0.634 

Wheel  control  input 

-0.680 

0.383 

0.568 

Bedford 

0.988 

Postflight  SWAT 

0.972 

Inflight  SWAT 

-0.301 

0.918 

IBI  variability 

0.955 

Eye  blink 

0.956 

VP 

3.842 

3.545 

1.639 

1.232 

The  above  factor  loading  matrix  has  been  rearranged  so  that  the  columns 
appear  in  decreasing  order  of  variance  explained  by  the  factors.  The  rows 
have  been  rearranged  so  that  for  each  successive  factor,  loadings  greater 
than  0.500  appear  first.  Loadings  less  than  0.25  have  been  blanked. 


H1 167 
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Our  interpretation  of  the  common  underlying  factors  can  be  summarized  as  follows: 

(a)  Measures  thought  to  reflect  physical  workload  load  highest  on  Factor  1  in 
the  PCA, 

(b)  Measures  reflecting  mental  workload  load  highest  on  Factor  2.  Factors  3 
and  4  do  not  account  for  much  of  the  variance  in  the  PCA, 

(c)  It  should  be  noted  that  Wheel  and  Column  control  inputs  load  on  Factor  3 
in  addition  to  Heart  Rate  Variability. 

9.5  DISCUSSION 

In  addition  to  the  criteria  of  validity  and  reliability,  this  contract  effort  has  used  replication 
in  order  to  further  give  confidence  to  the  interpretation  of  the  results.  The  Full-Mission 
simulation  utilized  the  lessons  learned  from  the  Part-Task  simulation  in  order  to  once 
again  subject  the  candidate  workload  measures  to  rigorous  empirical  scrutiny. 

As  was  done  with  the  Part-Task  simulation,  summaries  of  the  results  for  the  workload 
measures  are  provided.  The  first  table  summarizes  the  empirical  findings  of  validity  and 
reliability  (Table  9.5-1).  The  second  table  presents  the  rank  ordering  of  the  phases  of 
flight  for  each  workload  measure  (Table  9.5-2). 

9.5.1  DISCUSSION  OF  VALIDITY  AND  RELIABILITY  RESULTS 

To  summarize  the  results  of  two  comprehensive  simulation  studies  in  a  few  pages  is 
difficult.  Volume  I  was  written  with  the  intent  of  providing  as  much  of  the  empirical  results 
from  the  simulation  studies  as  possible.  It  is  an  overwhelming  task  to  attempt  to 
integrate  the  results  in  a  manner  that  would  state  a  clear  "winner"  as  to  which  is  the  best 
workload  measure  available.  Again,  we  suggest  that  the  reader  examine  the  results  for 
the  studies  one  workload  measure  at  a  time.  The  studies  were  designed  in  such  a 
fashion  that  the  measures  could  be  considered  as  if  it  were  the  only  dependent  variable 
in  the  simulation  effort.  In  this  fashion  a  reader  can  evaluate  a  given  workload  measure 
for  validity  and  reliability,  and  determine  if  the  results  were  replicated  from  the  Part-Task 
to  Full-Mission  simulation. 

SUBJECTIVE 

The  Full-Mission  simulation  examined  SWAT  both  In-Flight  and  Post-Flight  to  determine  if 
there  was  a  difference  for  probe  timing.  Although  no  main  effect  of  probe  timing  was 
found  (In-Flight  versus  Post-Flight)  a  significant  interaction  of  probe  timing  and  workload 
flight  (Nominal,  Communication,  and  Malfunction)  was  found.  The  overall  look  to  the 
probe  timing  difference  is  that  In-Flight  ratings  tend  to  be  lower  than  Post-Flight  ratings 
until  there  are  periods  of  high  workload  (malfunctions  for  example)  then  the  trend 
reverses.  The  reader  is  directed  to  a  paper  by  Corwin  (1989)  for  a  more  detailed 
discussion  of  the  In-Flight  versus  Post-Flight  results  from  the  Full-Mission  simulation. 

Both  SWAT  (In-Flight  and  Post-Flight)  and  the  Bedford  rating  scale  demonstrated 
evidence  of  validity  and  reliability.  The  Bedford  scale  was  the  first  time  a  single,  uni¬ 
dimensional,  scale  had  been  used  in  isolation  in  either  of  the  simulation  studies.  (The  1- 
to-20  point  Overall  Workload  Scale  was  used  in  conjunction  with  the  NASA-Task  Load 
Index  in  the  Part-Task  simulation.) 

The  Bedford  rating  scale,  similar  to  the  Modified  Cooper-Harper  or  McDonnell  Workload 
scale,  requires  no  data  reduction  as  do  SWAT  or  the  NASA-TLX.  The  elimination  of  the 
overhead  associated  with  the  "customization"  of  either  SWAT  or  the  NASA-TLX  makes 
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BDFD  -  BEDFORD  RATING  SCALE  HRV  -  STANDARD  DEVIATION  OF  IBI  (PER  MINUTE) 

SBP  -  POWER  SPECTRAL  ANALYSIS  PEDAL- PEDAL  CONTROL  INPUT 

(BLOOD  PRES.  COMPONENT)  (PER  MINUTE) 

SRS  -  POWER  SPECTRAL  ANALYSIS- 
(RESPIRATION  COMPONENT) 


Table  9.5-2 
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the  interpretation  of  the  data  a  much  more  straight  forward  exercise.  The  dividend  of  the 
subjective  techniques  based  on  multiple  bipolar  ratings  (SWAT  &  the  NASA-TLX)  is  the 
ability  to  examine  the  underlying  causes  of  workload  fluctuation  by  examining  the 
individual  bipolar  ratings.  The  examination  of  the  underlying  influences  on  workload  may 
or  may  not  be  better  addressed  in  a  aircraft  certification  effort  by  the  Pilot  Subjective 
Evaluation  (PSE)  technique  (Fadden,  1982;  Ruggiero  and  Fadden,  1987).  No  empirical 
evidence  exists  comparing  PSE  with  SWAT  or  the  NASA-TLX.  Yet,  the  PSE  directly 
addresses  the  Functions  and  Factors  of  FAR  25.1523  Appendix  D,  while  both  SWAT  and 
the  NASA-TLX  address  generic  underlying  factors. 

PHYSIOLOGICAL 

Eyeblink  rate  was  found  to  be  a  reliable  measure,  but  could  not  discriminate  among  the 
various  workload  conditions.  Again  the  reader  is  reminded  of  the  different  visual  tasks 
required  for  piloting  a  commercial  transport  aircraft,  some  which  may  cause  a  decrease 
in  eyeblink  rate  while  others  cause  an  increase.  This  may  be  the  reason  discriminability  * 

of  the  workload  conditions  was  not  found.  Eyeblink  rate  may  be  of  more  utility  in 
examining  workload  in  aircraft  environments  that  do  not  require  as  much  reading  and 
head  turning  to  scan  system  instruments,  such  as  tactical  fighter  aircraft. 

Mean  Heart  Rate  again  demonstrated  phenomenal  reliability:  all  25  test/retest  correlation  * 

coefficients  were  significant.  Unfortunately  mean  Heart  Rate  coulci  not  discriminate 
among  the  three  workload  flights  (Nominal,  Communication,  and  Malfunction).  The  lack 
of  discriminability  was  surprising.  In  an  attempt  to  understand  the  lack  of  discriminability 
Post  Hoc  a  careful  examination  was  made  of  the  changes  in  Heart  Rate  within  the 
measurement  windows  (Metalis  et.  al.,  1989).  It  appears  that  Heart  Rate,  whether 
influenced  by  arousal  and/or  workload,  is  extremely  sensitive  to  shifts  in  task  demands 
in  an  acute  fashion.  That  is,  the  phasic  changes  associated  with  Heart  Rate  occur  quite 
quickly  then  return  to  a  baseline  quickly.  The  interesting  point  to  note  is  the  length  of  the 
measurement  period  is  important  when  considering  workload.  A  measurement  window 
which  is  long  may  have  changes  in  Heart  Rate  due  to  increases  in  task  demands  mixed 
together  with  periods  of  rather  low  workload  (low  task  demands). 

Heart  Rate  Variability  again  demonstrated  poor  discriminab'1;*/  among  the  various 
workload  conditions.  Heart  Rate  Variability  did  not  demonstrate  evidence  of  reliability 
either. 

The  Blood  Pressure  and  Respiration  components  of  the  Power  Spectral  Analysis  did  not 
demonstrate  discriminability  in  the  Full-Mission  simulation.  The  Blood  Pressure 
component  did  however  demonstrate  reliability,  but  a  reliable  measure  that  cannot  * 

discriminate  among  different  workload  levels  does  not  have  much  utility. 

The  anomalous  findings  for  the  Respiration  component  in  the  Part-Task  simulation, 
increase  in  the  Respiration  component  with  increasing  workload,  was  not  replicated  in 
the  Full-Mission  simulation.  * 

PERFORMANCE 

Since  both  workload  measures  which  accompany  the  Secondary  Task  were  abandoned, 
only  the  control  input  activity  measures  were  examined  in  the  Full-Mission  simulation. 

As  was  demonstrated  in  the  Part -Task  simulation,  control  input  activity  for  the  Wheel  and 
Column  demonstrated  both  validity  and  reliability.  As  was  seen  in  the  Part-Task 
simulation  Pedal  activity  is  extremely  low  when  maneuvering  at  altitude.  Even  with  the 
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low  values  for  some  of  the  phases  of  flight  Pedal  activity  could  discriminate  between 
workload  flights  (Nominal,  Communication,  and  Malfunction).  The  low  values  in  some 
measurement  windows  had  a  detrimental  influence  on  the  reliability  coefficients  for  those 
phases  of  flight. 

9.5.2  DISCUSSION  OF  CORRELATION  MATRIX 

The  most  notable  relationship  change  from  the  Part-Task  to  the  Full-Mission  simulations 
is  the  significant  correlation  of  Heart  Rate  (IBI)  to  most  of  the  other  measures  (SWAT  in¬ 
flight  &  post-flight,  Bedford,  Wheel  &  Stick  control  inputs,  and  both  the  components  from 
the  power  spectral  analysis:  Blood  Pressure  &  Respiration). 

The  three  subjective  rating  methods  are  highly  inter-correlated.  An  interesting  finding  is 
that  the  highest  correlation  is  among  the  two  post-flight  measures  SWAT  (post-flight)  and 
Bedford  (r=0.98).  The  correlation  of  SWAT  in-flight  and  post-flight  is  smaller  (r=0.94) 
than  the  correlation  between  the  two  post-flight  measures. 

9 

The  anomaly  of  a  positive  correlation  of  the  respiration  component  (power  spectral 
analysis)  to  other  workload  measures  in  the  Part-Task  simulation  was  not  replicated  in 
the  Full-Mission  simulation. 

“  9.6  GENERAL  DISCUSSION 

The  contract  effort  has  attempted  to  identify  suitable  workload  assessment  techniques 
for  aircraft  certification  by  reviewing  the  literature  (Fact  Matrices),  find  consensus  among 
experts  (two  Workshops),  and  empirical  testing  (two  simulation  studies). 

The  experiments  we  do  today,  if  successful,  will  need 
replication  and  cross-validation  at  other  times  under  other 
conditions  before  they  can  become  an  established  part  of 
science,  before  they  can  be  theoretically  interpreted  with 
confidence.  (Campbell  and  Stanley,  1963) 

The  empirical  requirements  for  sound  research  put  forth  by  Campbell  and  Stanley  have 
been  complied  with  in  the  simulation  tests  conducted  as  part  of  this  contract.  A 
test/retest  methodology  was  employed  in  the  simulation  tests  to  determine  the  reliability 
of  each  measure  under  identical  test  conditions.  In  order  to  replicate  and  cross-validate 
the  results  two  simulation  tests  were  conducted.  The  second  simulation  test  included 
more  severe  workload  manipulations  (e.g.,  Missed  Approach;  "A,"  as  opposed  to  "B,” 
hydraulics  system  failure;  enroute  diversion),  as  well  as  simply  replicating  the  conditions 
'  of  the  first  simulation  test. 

Again,  it  should  be  mentioned  that  the  area  of  workload  assessment  is  a  developing 
science.  Subjective  ratings  (SWAT,  NASA-TLX,  Bedford,  1-to-20  Workload  score),  Heart 
Rate  (interbeat  interval),  and  Control  Input  Activity  all  demonstrated  evidence  of  validity 
-  and  reliability  in  the  simulation  studies  conducted  at  NASA-Ames.  Today's  results  will 

likely  become  obsolete  as  newer  methods  are  developed,  but  the  measures  identified 
can,  and  should,  be  used  to  help  answer  the  questions  of  the  amount  of  workload 
imposed  on  flightcrews. 
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10.0  CONTENTS  OF  VOLUME  TWO 

Perhaps  the  most  important  product  of  this  contract  is  Volume  Two  of  the  Final  Report. 
A  series  of  guidelines  were  developed  to  aid  in  the  evaluation  of  workload  assessment 
certification  programs  for  commercial  transport  aircraft. 

The  guidelines  were  complied  based  on  the  data  and  experience  obtained  in  this 
research  contract  from  the  literature  review,  part-task  simulation  testing,  fuil-mission 
simulation  testing,  and  the  two  workshops,  These  guidelines  include: 

(a)  Evaluation  criteria  for  assessment  techniques, 

(b)  Workload  assessment  techniques  guidelines, 

(c)  Guidelines  for  task  scenario  development. 

It  is  the  purpose  of  Volume  II  of  the  final  report  to  present  specific  guidelines  and 
recommendations  for  evaluating  workload  certification  plans.  No  attempt  is  being  made 
to  provide  a  "cookbook"  for  the  generation  of  an  aircraft  workload  certification  plan.  An 
emphasis  is  placed  upon  the  transient  nature  of  workload  assessment.  In  a  few  years, 
many  of  the  current  state-of-the  art  workload  measures  may  become  obsolete.  The 
contents  of  Volume  II  are  designed  to  allow  for  the  evaluation  of  current,  and  yet  to  be 
developed,  workload  assessment  techniques. 

Volume  Two  contains  guidelines  and  recommendations  for  evaluating  the  validity, 
reliability,  and  applicability  of  proposed  workload  certification  plans.  The  specific  areas 
addressed  for  evaluation  criteria  include: 

(a)  Validity, 

(b)  Reliability, 

(c)  Applicability. 

The  workload  assessment  techniques  are  broken  down  by  domain  area: 

(a)  Subjective, 

(b)  Performance, 

(c)  Physiological, 

(d)  Analytical. 

Advantages  and  liabilities  of  the  techniques  employed  in  the  simulation  studies  (as 
reported  in  Volume  One)  are  discussed.  Previous  work  reported  by  others  using  the 
various  assessment  techniques  is  documented  for  the  reader  as  well.  Additionally,  data 
derived  evidence  is  given  for  the  recommendation  of  specific  valid,  reliable,  and 
applicable  workload  measures. 

Finally,  the  process  of  evaluating  the  scenario  description  is  itemized: 

(a)  Scenario  Description 

(b)  Scenario  Evaluation  Criteria 

(c)  Relation  of  Workload  to  FAR  Requirements 

Since  most  aircraft  are  compared  to  so,.ie  "baseline"  aircraft  in  workload  certification,  the 
level  of  scenario  description  is  important  in  order  to  establish  a  common  ground  for 
comparison. 
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11.0  RECOMMENDATIONS  FOR  FUTURE  WORK 


The  present  work  does  not  presume  to  have  put  forth  the  definitive  empirical 
investigation  of  workload.  What  the  present  work  has  accomplished  is  a  rigorous  testing 
of  the  validity  and  reliability  of  existing  workload  measures  in  a  commercial  transport 
aircraft  environment.  It  is  hoped  that  this  work  will  serve  as  a  reference  to  personnel  who 
must  evaluate  the  adequacy  of  candidate  workload  measures  in  certification  plans  for 
FAR  25  type  approval. 

Follow-on  research  should  identify  the  points  (minimum  and  maximum)  for  each 
measure  where  crew  performance  breaks  down.  Peak  workloads  have  attracted  a  fair 
amount  of  attention  due  to  ever  increasing  demands  being  imposed  on  flight  crews. 
Current  design  practices,  however,  have  resulted  in  reduced  crew  workload  in  virtually 
every  segment  of  flight.  Currently  there  is  a  need  to  explore  low  workload.  In 
commercial  aviation,  underload  (long  periods  of  low  task-demands)  should  be  further 
explored  to  identify  the  affect  it  can  have  on  crew  performance.  If  the  crew  habituates  to 
a  lower  task  demands  (experiences  a  reduction  of  available  capacity)  due  to  underload, 
moderate  levels  of  task  difficulty  can  suddenly  be  experienced  as  high  workload. 

Another  pressing  issue  is  to  develop  a  technique  which  will  allow  the  results  of  workload 
assessment  from  different  pilots  to  be  evaluated  on  a  common  scale.  The  development 
of  an  "absolute"  workload  scale  may  not  be  so  much  a  question  of  workload 
methodology  as  it  is  psychometrics. 

Finally,  the  influence  of  workload  on  performance  needs  to  be  addressed  in  a  definitive 
fashion.  Intuitively  it  is  easy  to  understand  how  periods  of  high  workload  can  cause 
performance  to  drop  off.  The  subtle  influence  of  workload  on  performance  and  error 
rate  is  perhaps  the  behavioral  link  which  stimulates  the  interest  in  workload  in  an  aviation 
environment. 
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DISPATCH  RELEASE  PACKETS 


DISPATCH  RELEASES 


SFO-SCK 

FLIGHT  PATH 

Takeoff  SFO  28R 
Quiet  9  Departure 
REBAS  Intersection 
Direct  Modesto  VOR 
Landing  SCK  29R 


WEATHER 

SFO  CLR  20  59/45  2505  995 
SCK  CLR  20  62/40  2910  + 15  990 
SMF  CLR  15  60/52  1905  998 


ROUTE  DATA 


WEIGHT  &  BALANCE 


Distance  106  NM 

OEW 

101,600  lbs. 

Trip  Fuel  5,600  lbs. 

Payload 

23,400  lbs. 

Altitude  1 1 ,000  ft. 

ZFW 

125,000  lbs. 

ETE  0  +  27 

Fuel 

14,000  lbs. 

Alternate  SMF 

TOGW 

139,000  lbs. 

Distance  57  NM 

CG 

24% 

Fuel  2,500  lbs. 

Est.  LGW 

133,400  lbs. 

EIEO+14 

Holding  (30  Min.) 

3,700  lbs. 

Total  Fuel  Req. 

11,800  lbs. 

Fuel  on  board 

14,000  lbs. 

Reserves 

2,200  lbs. 

NOTAMS  MEL 


SCK  RWY  291  Closed  Autopilot  Inop. 


SMF-SFO 


FLIGHT  PATH 


WEATHER 


Takeoff  SMF  16 
Direct  SAC  VOR 
Risti  2  departure 
CEDES  Intersection 
Landing  SFO  28L 


SMF  CLR  15  60/52  1905  998 
SFO  CLR  20  59/45  2505  995 
OAK  CLR  10H  58/48  CLM  990 


ROUTE  DATA 


WEIGHT  &  BALANCE 


Distance  96  NM 

OEW 

101,600  lbs. 

Trip  Fuel  5,500  lbs. 

Payload 

23,400  lbs. 

Altftude  11,000  ft. 

ZFW 

125,000  lbs. 

ETE 0+25 

Fuel 

14,000  lbs. 

Alternate  OAK 

TOGW 

139,000  lbs. 

Distance  27  NM 

CG 

24% 

Fuel  1,800  lbs. 

Est.  LGW 

133,500  lbs. 
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ETE  0+10 
Holding  (30  Min.) 
Total  Fuel  Req. 

Fuel  on  board 
Reserves 

NOTAMS 

None 

LAX-(SFO)-(OAK)-SMF 

FLIGHT  PATH 

Takeoff  LAX  25R 
Ventura  9  Departure 
RZS  Transition 
J-501  BSR 
BSR  profile  descent 
SFO 

ROUTE  DATA 

Distance  290  NM 
Trip  Fuel  11 ,000  lbs. 
Altitude  FL310 
ETE 0+55 
Alternate  SMF 
Distance  85  NM 
Fuel  4,500  lbs. 

ETE 0+10 
Holding  (30  Min.) 
Total  Fuel  Req. 

Fuel  on  board 
Reserves 

NOTAMS 

None 


3,700  lbs. 
11,000  lbs. 
14,000  lbs. 
3,000  lbs. 

MEL 

Autopilot  Inop. 


WEATHER 


LAX  CLR  15  60/40  CLM  990 
SFO  CLR  20  59/45  2505  995 
SMF  CLR  15  60/52  1905  998 


WEIGHT  &  BALANCE 


OEW 

101 ,600  lbs. 

Payload 

23,400  lbs. 

ZFW 

125,000  lbs. 

Fuel 

25,000  lbs. 

TOGW 

150,000  lbs. 

CG 

21% 

Est.  LGW 

139,000  lbs. 

3,700  lbs. 

19,200  lbs. 

25,000  lbs. 

5,800  lbs. 

MEL 


Autopilot  Inop. 


* 
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APPENDIX  B 


FULL-MISSION  SIMULATION  EXAMPLE  SCENARIO 


FULL-MISSION  SIMULATION  EXAMPLE  SCENARIO 


FULL-MISSION  SIMULATION:  LONG  LEG  (HIGH  WORKLOAD) 
LAX  TO  SFO  DIVERTED  TO  OAK  DIVERTED  TO  SMF 

UPDATE  10/02/87 

SUNDAY,  JANUARY  25,  1987  TWILIGHT 
START  LEG  1850 


HH:MM:SS  CAPTAIN 


00:00:00  RECEIVES  TAKEOFF  CLEARANCE 

‘HEARS  TOWER, "  103,  CLEARED  FOR  TAKEOFF 

RUNWAY  24R. 


00:00:15  TAKEOFF 


*DEPRESSES  TOPS  OF  RUDDER  PEDALS  WITH  FEET. 

‘HEARS  BRAKE  LEVER  RELEASE. 

‘SENSES  AIRPLANE  START  TO  ROLL. 

‘PLACES  RIGHT  HAND  ON  THRUST  LEVERS. 

‘PLACES  LEFT  HAND  ON  OUTBOARD  GRIP  OF  CONTROL 
WHEEL. 

‘ADVANCES  THRUST  LEVERS  FOR  INITIAL  ACCELERATION  AND 
ALLOWS  EVEN  ENGINE  SPOOLUP. 

‘LOOKS  AT  EPR  INDICATORS  FOR  EVEN  ENGINE 
ACCELERATION. 

‘CONTINUES  THRUST  LEVERS  TO  APPROXIMATE  TAKEOFF 
SETTING 

‘CHECK  EPR  INDICATORS  FOR  TAKEOFF  BUG  SETTING 

‘KEEPS  RIGHT  HAND  ON  THRUST  LEVERS  AS  F/O  ADJUSTS. 

‘LOOKS  THROUGH  LEFT  FRONT  WINDOW  ALONG  RUNWAY 
CENTERLINE. 

‘STEERS  AIRPLANE  ALONG  RUNWAY  CENTERLINE 
WITHRUDDER  PEDALS. 

‘MAINTAINS  LIGHT  FORWARD  PRESSURE  ON  COLUMN. 

‘KEEPS  WINGS  LEVEL. 

‘HEARS  F/O,  "80  KNOTS.” 

‘LOOKS  AT  AIRSPEED  DISPLAY. 

‘SAYS,  "CHECK." 

‘CONTINUES  LOOKING  OUT  FORWARD  WINDOW. 


00:00:38  ROTATION 

‘HEARS  F/O,  "V  ONE." 

‘MOVES  RIGHT  HAND  FROM  THRUST  LEVER  TO 
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CONTROL  WHEEL 

♦HEARS  F/O,  "ROTATE." 

♦LOOKS  AT  AIRSPEED. 

(SEES  CORRELATION  WITH  F/O  REPORT). 

♦BEGINS  TO  APPLY  BACK  FORCE  ON  CONTROL  WHEEL 

♦ROTATES  TO  LIFTOFF  ATTITUDE. 

♦COMPLETE  ROTATION  TO  DESIRED  ATTITUDE  WITH 
REFERENCE  TO  ATTITUDE  INDICATOR. 

♦SENSES  LIFT  OFF. 

♦CHECKS  ALTIMETER  AND  RATE  OF  CLIMB  INDICATOR 
FOR  POSITIVE  RATE  OF  CLIMB. 

♦HEARS  F/O,  "VTWO.” 

♦HEARS  F/O,  POSITIVE  RATE  OF  CLIMB." 

♦CONFIRMS  INCREASING  ALTITUDE  AND  RATE  OF  CLIMB. 

00:00:47  GEAR  RETRACT  -  START  INITIAL  CLIMB 

♦CALLS,  "GEAR  UP." 

♦HEARS  F/O,  "GEAR  UP." 

♦SEES  F/O  COMPLIANCE  (PERIPHERAL  VISION). 

♦LOOKS  TO  SEE  IF  AIRSPEED  STABILIZED  AT 
APPROXIMATELY  V2  + 10. 

♦ADJUSTS  PITCH  ATTITUDE  TO  MAINTAIN  V2  + 10  IF 
NECESSARY. 

00:01 :05  NOISE  ABATEMENT  CLIMB 

♦MAINTAINS  V2  + 10  KNOTS  BY  ADJUSTING  PITCH 
ATTITUDE. 

00:01:40  FLAPS  15 

♦SEES  CLIMBING  THROUGH  2500  FEET  MSL. 

♦REDUCES  AIRPLANE  PITCH  ATTITUDE  TO  MAINTAIN 
500  TO  1000  FPM  VERTICAL  SPEED. 

♦SEES  AIRSPEED  INCREASING  ABOVE  V2  +  10 

♦CALLS,  "FLAPS  5." 

♦HEARS  F/O,  "FLAPS  5." 

♦SETS  AIRSPEED  CURSOR  TO  220  KNOTS. 

♦CONTINUES  PITCH  ADJUSTMENT  TO  MAINTAIN  500 
TO  1000  FPM. 

♦SETS  ELEVATOR  TRIM. 

00:01:50  FLAPS  5 


♦CHECKS  FLAP  INDICATOR  AT  5. 

♦CHECKS  AIRSPEED  ABOVE  160  KNOTS  ACCELERATING. 
♦CALLS,  "FLAPS  2." 

♦HEARS  F/O,  "FI  APS  2." 

♦CONTINUES  PITCH  ADJUSTMENT  TO  MAINTAIN  500 
TO  1000  FPM. 


00:01:51  LEVEL  AT  2500  FEET 


♦LEVELS  AIRPLANE  AT  2500  FEET  USING  ADI, 
ALTIMETER  AND  VERTICAL  SPEED  INDICATOR. 
♦ADJUSTS  THRUST  TO  MAINTAIN  REQUIRED  AIRSPEED. 
♦SETS  ELEVATOR  TRIM. 


00:02:12  FLAPS  2 

♦CHECKS  FLAP  INDICATOR  AT  2. 

♦CHECKS  AIRSPEED  AT  190  AND  ACCELERATING. 

♦CALLS,  "FLAPS  UP." 

♦CONTINUES  PITCH  ADJUSTMENT  TO  MAINTAIN  500 
TO  1000  FPM. 

♦SETS  ELEVATOR  TRIM 

00:02:26  PASSING  LOS  ANGELES  VORTAC  RADIAL  300. 

♦HEARS  LAX  TWR, "  103,  CONTACT  LAX  DEPARTURE 

ON  125.2. 

♦SEES  F/O  RETUNE  RADIO' 

♦HEARS  F/E  CALL  DEPARTURE. 

♦HEARS  LAX  DEP, "  103,  PROCEED  DIRECT  GORMAN, 

CLIMB  TO  FL310." 

♦HEARS  F/O, "  103,  DIRECT  GORMAN,  FL310. 

♦COMMANDS  F70,  "TUNE  GORMAN  ON  #1  NAV  RADIO." 
♦STARTS  RIGHT  TURN  AND  SETS  322  ON  COURSE  SELECTOR. 
♦SEES  AIRPLANE  ROLL  INTO  A  RIGHT  TURN. 


00:02:35  FU\PS  UP 


♦CHECKS  FLAP  INDICATOR  AT  0. 

♦CHECKS  AIRSPEED  AT  210  KNOTS  AND  ACCELERATING. 

♦CHECKS  LEADING  EDGE  FLAPS  TRANSIT  (AMBER)  LIGHT  OUT 
PRIOR  TO  EXCEEDING  210  KNOTS. 

♦CHECKS  FLAP  INDICATOR  AT  UP. 

♦MOVES  THROTTLES  UNTIL  3  EPRS  SHOW  CORRECT  SETTING 
FOR  CLIMB  POWER. 

♦HEARS  F/O,  "FLAPS  UP." 

♦ADJUSTS  PITCH  ATTITUDE  TO  ACCELERATE  TO  250  KNOTS.  * 

♦SETS  ELEVATOR  TRIM. 

00:02:40  COMPLETES  TURN 

♦SEES  AIRPLANE  ROLL  OUT  ON  HEADING  322  DEG.  * 

00:02:50  CUTBACK  THRUST 

♦SEES  AIRSPEED  APPROACHING  250  KNOTS. 

♦INCREASES  PITCH  ATTITUDE  SLIGHTLY  TO  MAINTAIN 
250  KNOTS. 

♦SEES  F/E  ADJUSTING  THRUST  LEVERS. 
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00:03:40  AFTER  TAKEOFF  CHECKLIST 

♦CALLS  "AFTER  TAKEOFF  CHECKLIST." 

♦HEARS  F/E,  "IGNITION,  OFF". 

♦HEARS  F/E,  "NO  SMOKING  AND  SEAT  BELT,  OFF." 

♦HEARS  F/E,  "ANTI-ICE  CLOSED,  GEAR  UP  AND  OFF,  FL<\PS 
UP,  NO  LIGHTS." 

♦HEARS  F/E,  "AUTO  PACK  TRIP  SWITCH,  CUT  OUT." 

♦HEARS  F/E,  "HYDRAULICS,  PRESSURE  AND  QUANTITY 
NORMAL" 

♦HEARS  F/E,  "PRESSURIZATION,  CHECKED  AND  SET". 
♦HEARS  F/E,  "AFTER  TAKEOFF  CHECKLIST  COMPLETE." 

00:04:40  CALLS  DISPATCH 

♦TELLS  F/O,  "GIVE  DISPATCH  TIMES." 

♦PRESSES  NO.1  VHF  RECEIVER  SWITCH  ON. 

♦HEARS  F/O, "  103,  SAN  FRANCISCO,  1 03  PUSH 

BACK  XXXX,  OFF  AT  XXXX,  ESTIMATE  SAN  FRANCISCO  XXX 
♦PRESSES  NO.1  VHF  RECEIVER  SWITCH  OFF. 

00:04:45  ENTERING  ICING  CONDITIONS 

♦OBSERVES  AIRCRAFT  APPROACHING  CLOUDS. 

♦OBSERVES  TAT  BELOW  10  DEGREES  C. 

♦TURNS  ON  ENGINE  IGNITION  AND  DIRECTS  F/O 
TURN  ON  ENGINE  ANTI-ICE. 

♦OBSERVES  EPR  DROP  ALL  3  ENGINES. 

♦OBSERVES  STABLE  ENGINE  OPERATION. 

00:04:48  WING  ANTI-ICE  ON 

♦OBSERVES  AIRCRAFT  ENTER  CLOUDS. 

♦OBSERVES  ICE  BUILDUP  ON  WINDSHIELD 
WIPERS. 

♦CALLS,  "WING  ANTI-ICE  ON." 

♦OBSERVES  EPR  DROP  ALL  THREE  ENGINES. 

♦COMMANDS  F/E  TO  RE-ACT  CLIMB  POWER 

00:05:00  PASSING  10,000  FEET 

♦OBSERVES  ALTIMETER  PASSING  10,000  FEET. 

00:05:20  OBSERVES  WEATHER  CONDITIONS 

♦OBSERVES  CLIMBING  ABOVE  OVERCAST  INTO  AN  AREA  OF 
CLEAR  AIR 

♦DIRECTS  F/O  TURN  OFF  ENGINE  AND  WING  ANTI-ICE. 
♦DIRECTS  F/E  TO  CHECK  CLIMB  POWER,  TURNS  OFF 
IGNITION  SWITCHES 

00:05:00  BLEED  TRIP 

♦HEARS  F/E,  "CAPTAIN,  I  HAVE  A  BLEED  TRIP 
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ON  NUMBER  3.  SWITCHING  AIR  SOURCE  TO  NUMBER  2." 

*CAPT  SAYS,  "ROGER,  LET  IT  COOL  AND  ATTEMPT  A  RESET. 

COMPLETE  THE  CHECKLIST." 

*F/E  ACKNOWLEDGES. 

00:06:30  CLEARANCE  TO  FL  310 

♦HEARS  DEPARTURE,  "BOEING  727  CLEARED  TO  FL310 
LOS  ANGELES  CENTER  ON  125.2." 

*HEARS  F/O,  "BOEING  727,  CLIMBING  TO  FL310,  LOS  ANGELES 
ON  125.2." 

♦SEES  F/O  SET  31,000  IN  ALTITUDE  SELECT  WINDOW 
♦OBSERVES  F/O  CHANGE  FREQUENCY  TO  125.2. 

00:06:45  RESUMES  CLIMB 

% 

♦SETS  AIRSPEED  CURSOR  TO  310  KNOTS. 

*  ADJUSTS  PITCH  ATTITUDE  TO  MAINTAIN  310  KNOT  CLIMB. 

♦SETS  ELEVATOR  TRIM. 

00:07:00  CONTACTS  LOS  ANGELES  CENTER  * 

♦HEARS  F/O,  "LOS  ANGELES  CENTER,  103  CLIMBING 
TOFL310." 

*HEARS  LOS  ANGELES  CENTER, "  103,  SQUAWK  IDENT." 

*HEARS  LOS  ANGELES  CENTER, "  103,  RADAR  CONTACT, 

MAINTAIN  FL  310  CONTACT  CENTER  ON  125.05  OUT  OF  230. 

00:08:00  TRANSITION  LEVEL  (18000  FEET) 

*SEES  APPROACHING  18,000  FEET. 

*CALLS,  "TRANSITION  LEVEL,  RESET  29.92." 

*RESETS  CAPTAIN'S  ALTIMETER  TO  29.92. 

*SAYS,  "ALTIMETER  SET." 

♦HEARS  F/O,  'TRANSITION  LEVEL,  RESET  29.92." 

00:12:00  CONTROL  CENTER  REPORT 

*HEARS  F/O,  LOS  ANGELES  CENTER,  103 

OUT  OF  230  FOR  310."  * 

♦HEARS  CENTER, "  103,  RADAR  CONTACT,  MAINTAIN  310. 


00:12:25  SPEED  REFERENCE  CHANGE 

*SEES  MACH  INDICATOR  IS  .78. 

♦USES  MACH  .78  AS  SPEED  REFERENCE  FOR  REMAINDER 
OF  CLIMB. 

00:15:00  CROSS  GORMAN  VORTAC 

♦SEES  OUTBOUND  COURSE  IS  304  DEGREE  TO  BIG  SUR 
♦SETS  HEADING  AND  COURSE  CURSORS  ON  HSI  FOR 
NEW  COURSE  OF  304  DEGREES 
♦OBSERVES  F/O  SETTING  BIG  SUR  114.0  ON 
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NAV  RADIO 

♦TURNS  TO  NEW  COURSE  OF  304  DEGREES 

♦SEES  VOR/LOC  LIGHT  IS  GREEN 

♦TELLS  F/O,  "SET  EPR  BUGS  TO  CRUISE  EPR." 

♦RETARDS  THRUST  LEVERS  SLIGHTLY. 

00: 1 7:00  ALTITUDE  ALERT 

♦SEES  ALTIMETER  APPROACHING  30,000  FEET. 

♦HEARS  ALTITUDE  ALERT. 

♦HEARS  F/O  "1000  FEET  TO  LEVEL  OFF." 

♦SEES  ALTITUDE  ALERT  LIGHT  ILLUMINATE. 

00:18:00  LEVEL  OFF  AT  FL  310 

♦SEES  ALTITUDE  ALERT  LIGHT  EXTINGUISH. 

♦LEVELS  AIRPLANE  AT  31 ,000  FEET  USING  ADI,  ALTIMETER  AND 
VERTICAL  SPEED  INDICATOR 

♦SEES  MACH  HAS  INCREASED  TO  .01  ABOVE  CRUISE 
MACH  SCHEDULE. 

♦PULLS  THROTTLES  BACK  SLIGHTLY. 

♦CALLS  "SET  CRUISE  THRUST." 

♦SEES  F/E  ADJUSTING  THRUST. 

♦SETS  ELEVATOR  TRIM. 

♦EXTRA  ATTENTION  MONITORING  THAT  ALTITUDE  SPEED  AND 
COURSE  ARE  STABILIZED. 

00:23:20  OBSERVES  DME 

♦OBSERVES  DME  SHOWING  DISTANCE  REMAINING  TO 
BIG  SUR  VORTAC. 

♦MENTALLY  CALCULATES  MINUTES  TO  ARRIVAL  OVER  BIG 
SUR. 

♦REMOVES  ENROUTE  CHART  FROM  UNDER  DEPARTURE 
PLATE. 

♦LOCATES  BIG  SUR  VORTAC  AND  NOTES  309  DEGREE  COURSE 
OUT  OF  BIG  SUR  TO  EUGEN  INTERSECTION. 

00:24:00  "A"  HYDRAULIC  PUMP  LOW  PRESSURE  LIGHT 

*  *HEARS  F/E,  "I  HAVE  A  LOW  PRESSURE  LIGHT  ON 

NUMBER  1  A  PUMP.  PRESSURE  AND  QUANTITY  ARE  OK." 
♦CAPT  SAYS,  "ROGER,  COMPLETE  THE  CHECKLIST." 

♦HEARS  F/E,  "PUMP  SWITCH  OFF.  I'LL  MONITOR 
PRESSURE  AND  QUANTITY." 

4  *CAPTAIN  ACKNOWLEDGES. 

00:31 :00  DIVERSION  SUMMARY 

♦HEARS  SELCAL  ACTIVATE. 

♦HEARS  F/O, "  103,  ANSWERING  SELCAL." 

♦HEARS  DISPATCH, "  103,  THIS  IS  SFO  DISPATCH.  SFO  IS 

CLOSED  DUE  TO  POWER  FAILURE.  REQUEST  YOU  DIVERT  TO 
OAKLAND  AND  WE'LL  BUS  THE  PASSENGERS  TO  SFO.  YOUR 
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ALTERNATE  REMAINS  SMF.  OAKLAND  WEATHER  IS  200  FT 
SCATTERED  400  FT  OVERCAST,  VISIBILITY  ONE  MILE  IN  FOG, 
TEMPERATURE  55,  DEWPOINT  54,  ALTIMETER  29.86. 

*HEARS  F/O,  “UNDERSTAND,  WELL  DIVERT  TO  OAK." 

00:33:00  REQUEST  CLEARANCE  TO  OAKLAND 

*CAPT  ORDERS  F/O  TO  REQUEST  CLEARANCE  TO  OAKLAND. 
♦HEARS  F/O,  "OAKLAND  CENTER,  103,  REQUESTING  TO 
DIVERT  TO  OAK." 

♦HEARS  OAK  CTR, "  103,  FROM  OVER  BIG  SUR,  CLEARED  TO 

CONTINUE  BIG  SURTROFILE  DESCENT  TO  MENLO,  EXPECT 
VECTORS  TO  THE  LOCALIZER  FOR  29R  AT  OAKLAND." 

♦HEARS  F/O, "  103,  FROM  BIG  SUR  PROFILE  DESCENT 

TO  MENLO,  VECTORS  TO  29R. 

00:34:00  CROSSING  BIG  SUR 

♦OBSERVES  #1  VOR  NEEDLE  SWING  CROSSING  BIG  SUR  VOR. 
♦SETS  COURSE  OF  309  IN  #1  COURSE  WINDOW. 

♦STARTS  LEFT  TURN  TO  MAINTAIN  COURSE  OF  309  DEG. 

00:34:35  ON  309  DEGREE  COURSE 

♦OBSERVES  AIRPLANE  ON  COURSE  OF  309  DEG. 

♦LEVELS  WINGS  TO  MAINTAIN  COURSE. 

00:36:00  TUNE  OAKLAND  VOR 

♦ORDERS  F/O  TO  TUNE  OAK  VOR  ON  #1  VOR. 

♦SETS  #1  COURSE  WINDOW  TO  331  DEG. 

♦NOTES  DME  DISTANCE  TO  CARME. 

00:36:20  REVIEWS  FLIGHT  PLAN 

♦LOOKS  AT  BIG  SUR  PROFILE  PLATE  IN  APPROACH  CHART 
HOLDER. 

♦REVIEWS  WAYPOINTS  AND  ALTITUDES. 

♦REVIEWS  ROUTE  DIAGRAM. 

♦RAISES  STAR  CHART  AND  LOOKS  AT  RWY  29  APPROACH 
CHART. 

♦REVIEWS  RWY  29  APPROACH  FREQUENCIES. 

♦LEAVES  CHART  UNDER  STAR  IN  CHART  HOLDER 
♦COMMANDS  F/O  TO  TUNE  OAK  VOR. 

♦SETS  331  IN  COURSE  WINDOW. 

♦CHECKS  DISTANCE  AND  ALTITUDE  RESTRICTIONS  ON 
DESCENT  CHART. 

00:36:50  ADVISES  CABIN  OF  DESTINATION  CHANGE 

♦ROTATES  MICROPHONE  SELECTOR  TO  SERVICE  INTERPHONE 
POSITION. 

♦PRESSES  ATTENDANT  CALL  SWITCH. 

♦HEARS  FLIGHT  ATTENDANT,  "FORWARD". 
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♦SAYS,  "WERE  DIVERTING  TO  OAKLAND  DUE  TO 
SAN  FRANCISCO  POWER  FAILURE.  ESTIMATE  ARRIVAL  IN 
OAKLAND  AT _ ." 

*HEARS  FLIGHTATTENDANT,  "OK,  THANKS,  I'LL  SPREAD  THE 
WORD." 

♦ROTATES  MICROPHONE  SELECTOR  TO  VHF2  POSITION. 

00:37:00  PASSING  CARME 

♦NOTES  COURSE  BAR  CENTERED  OVER  CARME. 

*TURNS  RIGHT  TO  MAINTAIN  COURSE  OF  331  DEG. 

*SETS  HEADING  BUG  TO  331  DEG. 

♦NOTES  COURSE  AND  HEADING  AGREE. 

♦LEVELS  WINGS  TO  MAINTAIN  331  DEG  COURSE. 

00:37:20  CHECKS  OAK  ATIS 

♦SELECTS  JEPPESEN  MANUAL  FROM  FLIGHT  KIT. 

*TURNS  TO  OAKLAND  SECTION  CHART  AND  AIRFIELD 
DIAGRAM  FROM  MANUAL. 

*SEES  OAKLAND  ATIS  FREQUENCY  IS  128.5. 

♦PLACES  CHARTS  IN  HOLDER  BENEATH  BIG  SUR  PROFILE 
DESCENT  PLATE. 

♦REMOVES  ILS  RWY  29  APPROACH  PLATES. 

♦PLACES  CHARTS  IN  HOLDER  BENEATH  BIG  SUR  ARRIVAL 
PLATE. 

♦REPLACES  MANUAL  IN  FLIGHT  CASE. 

♦SETS  128.5  IN  WINDOW  OF  NO.  1  VHF. 

♦PLACES  VHF  1  SWITCH  TO  ON. 

♦ADJUSTS  VOLUME  ON  NO.  1  VHF. 

♦ADVISES  F/O  "ATIS  ON  VHF  1." 

♦HEARS,  "OAKLAND  AIRPORT  TERMINAL  INFORMATION  GOLF. 
ESTIMATED  200  FT  SCATTERED,  400  FT  OVERCAST,  VISIBILITY 
ONE  MILE  IN  FOG,  WIND  CALM,  TEMPERATURE  55,  DEWPOINT 
54,  ALTIMETER  29.86.  DEPARTURES  AND  LANDINGS  ON 
RUNWAY  29.  INFORM  CONTROLLER  ON  INITIAL  CONTACT 
THAT  YOU  HAVE  INFO  GOLF." 

♦PLACES  VHF  1  SWITCH  TO  OFF. 

00:37:50  CONTROL  CENTER  CHANGE 

♦HEARS  CENTER, B  103,  CONTINUE  WITH  OAKLAND 
ON  125.2." 

♦HEARS  F/O, "  103,  'V  ;LAND  ON  125.2." 

♦HEARS  F/O,  OARLANl,  103,  FL310." 

♦HEARS  OAKLAND  CENTER^ "  103,  SQUAWK  IDENT." 

♦HEARS  OAKLAND  CENTER, "  103,  RADAR  CONTACT, 

MAINTAIN  FL  310. 

00:38:00  ADVISES  CABIN  OF  TOP  DESCENT 

♦ROTATES  MICROPHONE  SELECTOR  TO  SERVICE  INTERPHONE 
POSITION. 

♦PRESSES  ATTENDANT  CALL  SWITCH. 


*HEARS  FLIGHT  ATTENDANT,  "FORWARD." 

♦SAYS,  "STARTING  DESCENT.  ESTIMATING  OAKLAND 

♦ROTATES  MICROPHONE  SELECTOR  TO  VHF  2  POSITION. 

00:38:30  STARTS  DESCENT 

*  ADVISES  F/O  "TOP  DESCENT." 

*PULLS  THRUST  LEVERS  BACK  TO  IDLE. 

♦ESTABLISHES  DESCENT  USING  ADI,  AIRSPEED  AND  VERTICAL 
SPEED  INDICATORS. 

♦SETS  ELEVATOR  TRIM.  ( 

INTENSIFIES  INSTRUMENT^ SCAN  PATTER. 

♦HEARS  F/O,  "OAKLAND  CENTER,  103,  OUT  OF  FL  310." 

♦HEARS _ 103,  ROGER." 

00:39:03  CHECKS  DESCENT  PERFORMANCE 

♦DECIDES  TO  USE  .80M/320KIAS  SPEED  SCHEDULE  DURING 
DESCENT. 

♦ADJUSTS  PITCH  ATTITUDE  TO  MAINTAIN  .80M/320  KIAS. 

♦INCLUDES  DME  IN  INSTRUMENT  CROSS  CHECK. 

00:39:30  DESCENT/APPROACH  BRIEFING 

♦REPLACES  ARRIVAL  CHART  WITH  ILS  RWY  29  CHART  IN 
CHART  HOLDER. 

♦REVIEWS  APPROACH  CHART. 

♦SAYS,  "DESCENT  APPROACH  BRIEFING." 

♦SAYS,  “THIS  IS  A  BIG  SUR  PROFILE,  PRESENTLY  CLEARED  AS 
PER  PROFILE  DESCENT  TO  MENLO,  AN  ILS  TO  29,  108.7  ILS 
FREQUENCY,  INBOUND  COURSE  293  DEG  DECISION  HEIGHT 
205  MSL  AND  200  RADAR  ALTIMETER.  MISSED  APPROACH, 
CLIMB  TO  500  FEET,  THEN  CLIMBING  LEFT  TURN  TO  4000 
FEET  VIA  260  DEGREES  HEADING  OUTBOUND  TO  SAUSALITO 
VOR,  RADIAL  1 10  TO  SAUSALITO." 

♦SETS  200  FEET  ON  RADAR  ALTIMETER. 

♦SAYS,  "DECISION  HEIGHT  SET." 

♦HEARS  F/O,  "DECISION  HEIGHT  SET." 

00:40:15  REVIEWS  APPROACH  DATA 

♦RECALLS  DESTINATION  ENVIRONMENTAL  CONDITIONS  FROM 
ATIS  BROADCAST. 

♦LOOKS  AT  AIRFIELD  DIAGRAM  ON  APPROACH  CHART 
HOLDER. 

♦CHECKS  RUNWAY  LENGTH  ON  AIRFIELD  DIAGRAM. 

♦CHECKS  LANDING  WEIGHT  FROM  TOTAL  FUEL. 

♦CHECKS  PRESENT  VREF  FOR  FLAP  POSITIONS. 

♦DECIDES  ON  LANDING  FU\PS  30. 

♦TELLS  F/O,  "WELL  USE  FLAPS  30  FOR  LANDING." 

00:41:00  RECHECKS  WEATHER 

♦NOTES  LIGHT  TURBULENCE  HAS  BEGUN. 
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♦DECIDES  MODERATE  TURBULENCE  IS  POSSIBLE 
♦CALLS,  “SEAT  BELT  AND  START  SWITCHES  ON 
♦HEARS  F/O,  "SWITCHES  ON". 

00:42:00  TRANSITION  LEVEL  (18,000  FEET) 

♦SEES  APPROACHING  18,000  FEET. 

♦CALLS,  'TRANSITION  LEVEL,  RESET  29.86." 

♦SETS  CAPTAIN  ALTIMETER  AT  29.86." 

♦SETS  CAPTAIN  ALTIMETER  AT  29.86. 

♦HEARS  F/O,  TRANSITION  LEVEL,  RESET  29.86." 

♦COMPARES  F/O  ALTIMETER  WITH  CAPTAIN  ALTIMETER. 
♦SEES  ALTIMETERS  AGREE  WITHIN  50  FEET. 

♦SEES  FAILURE  FLAG  NOT  IN  VIEW  ON  CAPTAIN  ALTIMETER. 

00:42:40  SETS  AIR  SPEED  BUGS 

* 

♦RECEIVES  BUG  CARD  FROM  F/E. 

♦SETS  IAS  CURSOR  AT  VREF+5. 

♦SETS  1  BUG  AT  VREF  + 10 

4  *SETS  1  BUG  AT  200  K.. 

00:43:50  CROSS  SKUNK  INTERSECTION 

♦CALLS,  "SKUNK  INTERSECTION". 

♦SEES  VOR/LOC  LIGHT  IS  GREEN. 

00:43:40  RECHECKS  DESCENT  PERFORMANCE 

♦LOOKS  AT  DME  DISTANCE  FROM  OAKLAND  VOR. 

♦LOOKS  AT  VERTICAL  SPEED  INDICATOR. 

♦DECIDES  DESCENT  PERFORMANCE  IS  PROBABLY 
SATISFACTORY. 

00:45:00  LEVEL  OFF  AT  10,000  FEET  AND  CROSS  BOLDR  INTERSECTION 

♦SEES  OAK  DME  34  MILES. 

♦SEES  ALTITUDE  ALERT  LIGHT  EXTINGUISH. 

♦LEVELS  AIRPLANE  AT  10,000  FEET  USING 
ALTIMETER  AND  VERTICAL  SPEED  INDICATOR. 

*  *ADVANCES  THRUST  TO  MAINTAIN  250  KNOTS. 

♦SETS  ELEVATOR  TRIM. 

♦CALLS,  "LANDING  LIGHTS  ON." 

♦SEES  F/O  COMPLIANCE. 

♦CALLS,  "BOLDR  INTERSECTION". 

4  *SEES  331  DEG  COURSE  CONTINUES  TO  MENLO 

INTERSECTION. 

00:45:10  HAND  OFF  TO  APPROACH  CONTROL 

♦HEARS  CENTER, "  103,  15  MILES  FROM  MENLO,  CONTACT 

OAKLAND  APPROACH  124.4." 

♦HEARS  F/O, "  1 03,  GOING  TO  1 24.4. 

♦HEARS  F/0,  "OAKLAND  APPROACH,  THIS  IS  103,  WITH  YOU 
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AT  10  000  M 

♦HEARS  OAK  APP, "  1 03,  RADAR  CONTACT,  MAINTAIN 

10,000." 

♦HEARS  F/O, H _ 103,  ROGER." 

00:48:00  CROSS  MENLO  INTERSECTION 

♦SEES  OAK  DME  16  MILES. 

♦CALLS  "MENLO  INTERSECTION" 

♦FOLLOWS  FD  TO  ROLL  OUT  ON  348  DEGREES. 

00:48:15  CHANGES  AND  CHECKS  APPROACH  STATUS 

♦MOVES  ILS  RWY  29  APPROACH  CHART  TO  TOP  POSITION  IN 
HOLDER. 

♦REVIEWS  KEY  ITEMS  OF  OAKLAND  ILS  RWY  29  APPROACH  - 
FREQUENCIES,  COURSE,  MINIMUM  SECTOR  ALTITUDE,  FIELD 
ELEVATION,  INITIAL  APPROACH  ALTITUDE,  OUTER  MARKER 
CROSSING  ALTITUDE,  DECISION  HEIGHT,  MISSED  APPROACH 
PROCEDURE. 

♦PLACES  MARKER  BEACON  AUDIO  SWITCH  ON. 

♦PLACES  MARKER  SWITCH  ON  HIGH. 

♦NOTES  NO  MARKER  AUDIO  SIGNAL. 

00:49:00  DESCENT  CLEARANCE  TO  5000 

♦HEARS  APPROACH, "  103,  DESCEND  AND  MAINTAIN  5000, 

FLY  HEADING  010  TOTRTERCEPT  THE  29R  LOCALIZER." 
♦HEARS  F/O, "  1 03,  OUT  OF  TEN  FOR  FIVE,  01 0  TO  29R 

LOCALIZER  " 

♦SEES  F/O  SET  5,000  IN  ALTITUDE  SELECT  WINDOW. 

00:49:10  STARTS  DESCENT  TO  5000 

♦PULLS  THRUST  LEVERS  BACK  TO  IDLE. 

♦ESTABLISHES  DESCENT  USING  ADI,  AIRSPEED  AND  VERTICAL 
SPEED  INDICATORS. 

♦SETS  ELEVATOR  TRIM. 

00:50:00  LEVEL  AT  5000  FEET 

♦SEES  ALTITUDE  ALERT  LIGHT  EXTINGUISH 
♦LEVELS  AIRPLANE  AT  5000  FEET  USING  ADI,  ALTIMETER  AND 
VERTICAL  SPEED  INDICATOR 

♦OBSERVES  DISTANCE  TO  OAKLAND  VORTAC  ON  DME 
INDICATOR. 

00:50:15  DESCENT  APPROACH  CHECKLIST 

♦CALLS,  "DESCENT  APPROACH  CHECKLIST." 

♦HEARS  F/E,  "SEAT  BELT,  ON". 

♦HEARS  F/E,  "ANTI-ICE,  CLOSED." 

♦HEARS  F/E,  "LANDING  LIGHTS,  ON". 

♦HEARS  F/E,  "ALTIMETERS,  SET  AND  CROSS  CHECKED". 
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*"RADAR  ALTIMETER",  F/0  "CHECKED". 

♦"FLIGHT  INSTRUMENTS,  FDS,  AND  RADIOS",  F/O  "SET  AND 
CROSS  CHECKED". 

♦"GO-AROUND  EPR  AND  VREF"  F/O  "BUGS  SET. 

♦HEARS  F/E,  "FUEL,  SET  FOR  LANDING." 

♦HEARS  F/E,  "HYDRAULICS,  PRESSURE  AND  QUANTITIES 
NORMAL". 

♦HEARS  F/E,  "PRESSURIZATION  AND  COOLING  DOORS", 

♦SET". 

♦HEARS  F/E,  "CIRCUIT  BREAKERS,  CHECKED". 

♦HEARS  F/E,  "DESCENT  APPROACH  CHECKLIST  COMPLETE." 

00:51 :00  DESCENT  CLEARANCE  TO  2500 

♦HEARS  APPROACH, 11  103,  DESCEND  TO  2500." 

♦HEARS  F/O, "  1 03, "OUT  OF  FIVE  FOR  2500." 

#  *TELLS  F/O,  "ATTITUDE  SELECT  3000." 

♦SEES  F/O  COMPLIANCE. 

00:52:00  TUNES  ILS  FOR  APPROACH 

4  *TELLS  F/O,  "TUNE  THE  ILS." 

♦SETS  COURSE  CURSOR  ON  293  DEGREES. 

*HEARS  F/O,  "YOU  HAVE  ILS  IDENTIFIED." 

00:52:10  REDUCES  AIRSPEED 

♦ADJUSTS  PITCH  ATTITUDE. 

♦LETS  AIRSPEED  DROP  TO  200., 

♦SETS  ELEVATOR  TRIM. 

00:52:15  ALTITUDE  ALERT 

♦SEES  APPROACHING  3500  FEET. 

♦HEARS  ALTITUDE  ALERT. 

♦HEARS  F/O,  "1000  FEETTO  LEVEL  OFF." 

♦SEES  ALTITUDE  ALERT  LIGHT  ILLUMINATE. 

00:52:25  FLAPS  2 

*  *CALLS,  "FLAPS  2." 

♦HEARS  F/O,  "FLAPS  2." 

♦HEARS  GEAR  WARNING  HORN. 

*F/E  PULLS  HORN  CUTOUT  SWITCH. 

♦SETS  ELEVATOR  TRIM. 

•  *SEES  FLAP  INDICATOR  AT  FLAPS  2  AND  ANNUNCIATOR  LIGHT 

GREEN. 

♦ADJUSTS  PITCH  ATTITUDE  AND  SLOWS  AIRPLANE  TO 
190  KNOTS. 

00:52:45  FLAPS  5 

♦SEES  AIRSPEED  IS  AT  190  KNOTS. 

♦CALLS,  "FLAPS  5." 
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♦HEARS  F/O,  "FLAPS." 

♦SETS  ELEVATOR  TRIM. 

♦SEES  FLAP  INDICATOR  AT  FLAPS  5  AND  ANNUNCIATON 
LIGHT  GREEN. 

♦SLOWS  AIRPLANE  TO  160  KNOTS. 

00:53:00  LEVEL  OFF  AT  2500  FEET 

♦SEES  ALTITUDE  ALERT  LIGHT  EXTINGUISH. 

♦LEVELS  AIRPLANE  AT  2500  FEET  USING  ADI,  ALTIMETER  AND 
VERTICAL  SPEED  INDICATOR. 

♦PLACES  FD  ALTITUDE  HOLD  SWITCH  ON. 

♦ADVANCES  THRUST  TO  MAINTAIN  150  KNOTS. 

♦SETS  ELEVATOR  TRIM. 

00:53:15  FLAPS  15  * 

♦SEES  AIRSPEED  IS  AT  160  KNOTS. 

♦CALLS,  "FLAPS  15." 

♦HEARS  F/O,  "FLAPS  15." 

♦SETS  ELEVATOR  TRIM.  * 

♦SEES  FLAP  INDICATOR  AT  FLAPS  15  AND  ANNUNCIATOR 
LIGHT  GREEN. 

♦SLOWS  AIRPLANE  TO  150  KNOTS. 

00:53:30  ILS  APPROACH  CLEARANCE 

♦HEARS  APPROACH, "  103,  CLEARED  TO  INTERCEPT 

LOCALIZER,  MAINTAfN"2500  UNTIL  ESTABLISHED 
INBOUND.  CLEARED  FOR  ILS  29,  CONTACT  OAKLAND  TOWER 
127.2  AT  THE  MARKER. 

♦HEARS  F/O, " _ 103,  TOWER  127.2  AT  THE  MARKER. 

00:54:00  LOCALIZER  ALIVE 

♦HEARS  F/O,  "LOCALIZER  ALIVE." 

♦SEES  LOCALIZER  MOVING  TOWARD  CENTER  OF  HSI  SCALE. 

00:54:05  STARTS  TURN  TO  INTERCEPT  LOCALIZER 

♦SETS  HEADING  CURSOR  TO  293  DEGREES.  * 

00:54:30  COMPLETES  TURN 


♦SEES  AIRPLANE  ROLL  OUT  ON  LOCALIZER. 
00:55:00  LOCALIZER  CAPTURE 
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♦SEES  HEADING  APPROACHING  293  DEGREES. 

♦SEES  FD  ANNUNCIATORS  AGREE. 

♦SEES  AIRPLANE  ROLLING  OUT  OF  TURN. 

♦SEES  LOCALIZER  IN  HSI  IS  CENTERED. 

♦SEES,  FD  VOR/LOC  LIGHT  GREEN,  GLIDE  SLOPE  LIGHTS 
AMBER. 
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00:55:15  SETS  MISSED  APPROACH  DATA 


♦REVIEWS  MISSED  APPROACH  PROCEDURE  ON  APPROACH 
CHART. 

♦SETS  HEADING  CURSOR  ON  260  DEGREES. 

♦TELLS  F/O,  "ALTITUDE  SELECT  4000." 

♦HEARS  F/O,  "4000." 

00:55:40  GLIDE  SLOPE  ALIVE 

♦HEARS  F/O,  "GLIDE  SLOPE  ALIVE." 

♦SEES  GLIDE  SLOPE  MOVING  TOWARD  CENTER  OF  ADI  AND 
HSI. 

♦CALLS,  "GEAR  DOWN." 

♦SEES  F/O  LOWER  GEAR. 

♦NOTES  LANDING  GEAR  LIGHTS  3  GREEN. 

♦SEES  AIRPLANE  IS  ONE  DOT  BELOW  GLIDESLOPE. 

♦CALLS,  "FLAPS  25." 

♦NOTES  F/O  SET  FLAPS. 

♦REDUCES  AIRSPEED  TO  140  KNOTS. 

♦TRIMS  ELEVATOR. 

00:56:00  GLIDE  SLOPE  CAPTURE 

♦SEES  AIRPLANE  IS  ON  GLIDESLOPE. 

♦CALLS,  "FLAPS  30,  LANDING  CHECKLIST." 

♦HEARS  F/E,  "ANTI-SKID,  5  RELEASES." 

♦HEARS  F/E,  "IGNITION  ON." 

♦HEARS  F/E,  "NO  SMOKING,  ON." 

♦HEARS  F/E,  “GEAR,  DOWN,  IN,  3  GREEN." 

♦HEARS  F/E,  "FLAPS  40,  40,  GREEN  LIGHT." 

♦HEARS  F/E,  "HYDRAULICS,  OK." 

♦HEARS  F/E,  "LANDING  CHECKLIST  COMPLETE." 

♦CAPTAIN  ACKNOWLEDGES. 

♦REDUCES  AIRSPEED  TO  V  REF  +  30. 

♦TRIMS  ELEVATOR. 

00:57:30  EXECUTE  MISSED  APPROACH 

♦REACHES  MINIMUMS.  F/O  REPORTS  "NO  RUNWAY". 

♦ADD  POWER 
♦ROTATE 

♦CALLS  FOR  FLAPS  25 
♦NOTES  POSITIVE  CLIMB  RATE 
♦CALLS  FOR  GEAR  UP 

00:59:35  ENGINE  OUT 

*F/E  REPORTS,  "NO.  3  ENGINE  FLAMED  OUT’. 

♦CAPT  ACKNOWLEDGES 
♦CAPT  ADDS  RIGHT  RUDDER 

♦CAPT  CALLS,  "ENGINE  FAILURE/SHUTDOWN  CHECKLIST 
♦CAPT  TRIMS  RUDDER  AND  STABILIZER 
♦HEARS  F/O,  "ESSENTIAL  POWER  ON  OPERATING 


GENERATOR." 

♦CAPT  DIRECTS  F/O  TO  DECLARE  AN  EMERGENCY 
*F/0  DECLARES  EMERGENCY  AND  REQUESTS  RADAR 
VECTOR  DIRECT  TO  SMF 

01:00:00  PASSING  500  FEET 

♦CAPT  OBSERVES  ALTIMETER  PASSING  500  FEET 
♦INITIATES  CLIMBING  LEFT  TURN  TO  260  DEG  HEADING 
TOWARD  4000  FEET  DIRECT  TO  SAUSALITO. 

01:00:30  ENGINE  FAILURE/SHUTDOWN  CHECKLIST 

*F/E  CALLS,  THRUST  LEVER  NO.  3,  CLOSE" 

♦CAPT  ACKNOWLEDGES  AND  CLOSES  THRUST  LEVER 

*F/E  CALLS,  "START  LEVER,  CUTOFF"  * 

♦CAPT  ACKNOWLEDGES  AND  CLOSES  START  LEVER. 

*F/E  CALLS,  "GALLEY  POWER,  OFF". 

*F/E  CALLS,  "CARGO  OUTFLOW  VALVE,  CLOSED". 

*F/E  CALLS,  "PACK  SWITCH,  OFF". 

*F/E  CALLS,  "GENERATOR  BREAKER  LIGHT,  ON".  * 

*F/E  CALLS,  "ELECTRICAL  LOAD,  MONITORED". 

*F/E  CALLS,  "FUEL  SHUTOFF  SWITCH,  CLOSED". 

*F/E  CALLS,  "ENGINE  BLEED  SWITCH,  CLOSED". 

*F/E  CALLS,  "WING  AND  ENGINE  ANTI-ICE,  NOT  REQUIRED". 

*F/E  CALLS,  "ONE  GENERATOR  INOP  CHECKLIST  IS 
COMPLETED;  ENGINE  FAILURE  CHECKLIST  COMPLETE". 

♦CAPT  ACKNOWLEDGES 

01:00:50  NO  RESTART  ATTEMPTED 

*F/E  INFORMS  CAPTAIN  THAT  A  RESTART  SHOULD  NOT  BE 
ATTEMPTED  DUE  TO  NO  N1  COMPRESSOR  ROTATION. 

*CAPT  ACKNOWLEDGES. 

01 :01 :00  FLAP  RETRACTION  AND  CLEANUP 

♦CAPT  OBSERVES  AIRSPEED  AT  V2+ 10. 

*CAPT  CALLS,  "FLAPS  15". 

♦ACCELERATES  TO  1 50  / 

♦ORDERS  F/O  TO  TUNE  SAUSALITO  VORTAC. 

♦CAPT  OBSERVES  FLAPS  15  AND  150  KNOTS 
♦CAPT  CALLS,  "FLAPS  5". 

♦ACCELERATES  TO  160. 

♦CAPT  OBSERVES  AIRSPEED  1 60.  • 

♦CAPT  CALLS,  "FLAPS  2". 

♦ACCELERATES  TO  190. 

♦OBSERVES  AIRSPEED  190  KNOTS. 

♦CAPT  CALLS,  "FLAPS  UP". 

♦ACCELERATES  TO  200. 

♦CAPT  ADJUSTS  PITCH,  TRIM  AND  STABILIZER. 
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01 :01 :30  LEVEL  4000  FEET,  FLAP  RETRACTION  COMPLETE 

*SEES  ALTITUDE  ALERT  LIGHT  EXTINGUISH 
*LEVELS  AIRPLANE  AT  4000  FEET  USING  ADI,  ALTIMETER  AND 
VERTICAL  SPEED  INDICATOR. 

*HEARS  F/O,  "FLAPS  UP". 

01 :03:00  INTERCEPT  110  DEGREE  COURSE  SAUSALITO  VORTAC 

*SEES  OUTBOUND  HEADING  FROM  SAUSALITO  VORTAC  IS 
290  DEG." 

*SETS  HEADING  AND  COURSE  CURSORS  ON  HSI  FOR  NEW 
COURSE  OF  290  DEGREE. 

01 : 03:45  REQUESTS  EMERGENCY  DIVERT  TO  SACRAMENTO  METRO 

*  *CALLS  OAKLAND  CENTER, "  103  ON  MISSED  APPROACH 

FROM  OAKLAND,  ENGINE  OUT7REQUESTING  EMERGENCY 
DIVERT  TO  SACRAMENTO".  REQUEST  7000  FEET. 

*HEARS  CENTER, "  103,  SMF  REPORTING 

CLEAR  AND  15,  WHEN  PASSING  4000  FEET  TURN  RIGHT  035 
4  DEGREES  AND  PROCEED  DIRECT  SACRAMENTO. 

MAINTAIN  7000." 

*HEARS  F/O, "  103,  ROGER,  RIGHT  035  AT  4000,  DIRECT 

SACRAMENTOT7000." 

01 :05:45  *ORDERS  F/O  TO  ADVISE  DISPATCH  OF  DIVERSION  TO  SMF. 

01 :06:30  ALTITUDE  WARNING 

*  HEARS  ALTITUDE  WARNING. 

*HEARS  F/O,  "1000  FT  TO  LEVEL  OFF." 

*SEES  ALTITUDE  ALERT  LIGHT  ILLUMINATE. 

*CAPT  ACKNOWLEDGES. 

01:07:30  LEVELS  AT  7000 

*SEES  ALTITUDE  ALERT  LIGHT  EXTINGUISH. 

*NOTES  ALTIMETER  AT  7000  FEET. 

*LEVELS  AIRCRAFT  AT  7000  FEET  USING  ADI,  ALTIMETER  AND 
4  VERTICAL  SPEED  INDICATOR. 

*REDUCES  POWER  TO  CRUISE  SETTING. 

*SETS  AIRSPEED  CURSOR  TO  288  KNOTS. 

*NOTES  AIRSPEED  STABLE  AT  2  ENGINE  CRUISE. 

*  01 :08:45  "A"  HYDRAULICS  SYSTEM  FAILURE 

*F/E  REPORTS,  "CAPT,  A  QUANTITY  IS  DECREASING  BELOW 
2.5,  A  PUMP  LIGHTS  ARE  ON.  WE’VE  JUST  LOST  "A”  SYSTEM". 
*CAPT  CALLS,  "A"  SYSTEM  FAILURE  CHECKLIST 
*F/E  CALLS,  "CONTROL  WHEEL  NEUTRAL". 

*CAPT  REPORTS,  "NEUTRAL"  AND  NEUTRALIZES  CONTROL. 
*F/E  CALLS,  "SPOILER  SWITCH  SYSTEM  A,  OFF" 

*CAPT  TURNS  SPOILER  SWITCH  OFF. 
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*F/E  CALLS,  "STANDBY  RUDDER  SWITCH,  ON". 

*CAPT  TURNS  STANDBY  RUDDER  SWITCH  ON. 

*F/E  CALLS,  "SYSTEM  A  PUMP  SWITCH,  OFF". 

*CAPT  ACKNOWLEDGES. 

*F/E  CALLS,  "AUTOPILOT  ELEVATOR  SERVO  SYSTEM  B". 

*CAPT  PLACES  AUTO  PILOT  SI  SERVO  TO  B  POSITION. 

*F/E  CALLS,  "SYSTEM  A  FLUID  SHUTOFF  SWITCH,  CLOSE". 

*F/E  READS,  "REVIEW  SYSTEM  "A"  LOSS  ADVISORY  ITEMS. 
*CAPT  SAYS,  "CONTINUE". 

*F/E  READS,  "CHECK  WEATHER,  CROSSWIND  LIMIT  19  KNOTS. 
OBSERVE  YAW  DAMPER  LIMITATION." 

*F/E  READS,  "PLAN  MANUAL  GEAR  AND  ALTERNATE  FLAP 
EXTENSION." 

*F/E  READS,  "PLAN  FLAPS  15  LANDING,  USE  VREF+15  KNOTS. 
OUTBOARD  AND  GROUND  SPOILERS,  NOSE  WHEEL  STEERING 
IS  INOPERATIVE." 

*F/E  READS,  “DO  NOT  OPEN  GROUND  INTERCONNECT  AFTER 
LANDING.” 

*CAPTAIN  ACKNOWLEDGES. 

01 : 18:45  CLEARANCE  TO  2600  FEET 

*HEARS  CENTER, "  103,  DESCEND  AND  MAINTAIN 

2600  FEET". 

*  HEARS  F/O,  "BOEING  727,  OUT  OF  SEVEN  FOR  2600. 

*SEES  F/O  SET  2600  FEET  IN  ALTITUDE  SELECT  WINDOW 
*TELLS  F/O  TO  HAVE  CENTER  REQUEST  THAT  SMF  HAVE 
EMERGENCY  EQUIPMENT  STANDING  BY  DUE  TO  ENGINE  AND 
HYDRAULIC  FAILURE,  AND  TO  RELAY  NUMBER  OF 
PASSENGERS  ABOARD  AND  TOTAL  FUEL  ON  BOARD. 


01:18:50  DESCENT 


*CAPT  CALLS,  "DESCENT  AND  APPROACH  CHECKLIST  FOR 
"A"  SYSTEM  FAILURE  DOWN  TO  FLAP  EXTENSION,  AND 
REVIEW  ONE  ENGINE  INOPERATIVE  DESCENT  AND 
APPROACH  CHECKLIST." 

*F/E  CALLS,  "PRESSURIZATION  AND  COOLING  DOORS,  SET." 
*F/E  CALLS,  "SEAT  BELT,  ON" 

*F/E  CALLS,  "ANTI-ICE" 

*CAPT  RESPONDS,  "NOT  REQUIRED". 

*F/E  CALLS,  "ALTIMETERS,  SET  AND  CROSS  CHECKED". 
*CAPT  AND  F/O  RESPOND,  "SET  AND  CROSS  CHECKED". 
*F/E  CALLS,  "FLIGHT  INSTRUMENTS,  RADIOS,  AND  FDS, 

SET  AND  CROSS  CHECKED". 

*CAPT  AND  F/O  RESPOND,  "SET  AND  CROSS  CHECKED". 
*F/E  CALLS,  "FLAP  INHIBIT  SWITCH,  INHIBIT". 

*CAPT  OBSERVES  F/O  PUT  SWITCH  TO  INHIBIT 

*F/E  CALLS,  "GO-AROUND  EPR  AND  VREF  +  15,  SET  BUGS". 

*CAPT  SETS  HIS  AIRSPEED  BUG 

*F/E  CALLS,  "FUEL,  SET  FOR  LANDING". 

*F/E  CALLS,  "CIRCUIT  BREAKER,  CHECK". 

*F/E  CALLS,  "ALTERNATE  FLAP  MASTER  SWITCH,  ON". 

*CAPT  OBSERVES  F/O  TURN  SWITCH  ON". 
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♦F/E  CALLS,  "FLAP  LEVER,  15". 

*CAPT  OBSERVES  F/O  PLACE  FLAP  HANDLE  TO  15 
*F/E  CALLS,  "COMPLETE  TO  FLAP  AND  GEAR  EXTENSION. 

THE  ONLY  ITEM  OF  INTEREST  ON  THE  ENGINE  INOP 
CHECKLIST  IS  TURNING  THE  RIGHT  PACK  SWITCH  OFF. 

I’LL  DO  IT  NOW.  WE  SHOULD  USE  FLAPS  15  FOR  LANDING. 
♦CAPT,  "ROGER,  TURN  IT  OFF,  ROGER,  FLAPS  15 
FOR  LANDING". 

*F/E  CALLS,  "STANDBY  FOR  MANUAL  GEAR  EXTENSION" 
♦CAPT,  "STANDBY". 

01:19:20  APPROACH  CLEARANCE 

♦HEARS  CENTER, "  103,  FROM  OVER  SAC  FLY  HEADING  310 

TO  INTERCEPT  THETOCALIZER  FOR  RUNWAY  34.  CLEARED 
FOR  THE  APPROACH.. 

*  HEARS  F/O,  "ROGER,  VECTOR  310  DEGREES,  UNDERSTAND 
CLEARED  ILS  RUNWAY  34." 

♦ORDERS  F/O  TO  TUNE  AND  IDENTIFY  THE  ILS  ON  #1  RADIO. 
*SETS  COURSE  WINDOW  TO  344  DEGREES. 

01:19:50  GEAR  AND  FLAP  EXTENSION 

♦CAPT  REDUCES  POWER,  TRIMS  RUDDER  AND  STABILIZER. 
*CAPT  NOTES  AIRSPEED  IS  200  KNOTS. 

*CAPT  CALLS,  "EXTEND  FLAPS  TO  15." 

*CAPT  NOTES  F/O  STARTING  FLAP  EXTENSION. 

*CAPT  CALLS,  "GEAR  HANDLE  OFF." 

*CAPT  NOTES  HANDLE  IS  OFF. 

*CAPT  CALLS,  "EXTEND  THE  GEAR  MANUALLY." 

*CAPT  NOTES  F/E  STARTING  GEAR  EXTENSION. 

*CAPT  NOTES  FLAPS  AT  2  DEG,  AIRSPEED  190  KNOTS. 

♦CAPT  NOTES  FLAPS  AT  5  DEG,  AIRSPEED  160  KNOTS. 

♦CAPT  NOTES  FLAPS  AT  15  DEG,  AIRSPEED  150  KNOTS. 

01 :20:20  CROSSING  SAC  VOR 

*STARTS  LEFT  TURN  TO  310  DEG  HEADING  TO  INTERCEPT 
LOCALIZER. 

01 :21 :00  LANDING  CHECKLIST 

♦CALLS,  "LANDING  CHECKLIST,  "A"  SYSTEM  INOP." 

♦HEARS  F/E,  "GPWS,  OFF." 

♦HEARS  F/E,  "NO  SMOKING,  ON." 

♦HEARS  F/E,  "IGNITION,  ON." 

♦HEARS  F/E,  "GEAR,  DOWN,  IN,  3  GREEN." 

♦HEARS  F/E,  "ANTI-SKID,  5  RELEASES." 

*  HEARS  F/E,  "FLAPS  15,  15,  GREEN  LIGHT." 

♦HEARS  F/E,  "HYDRAULICS,  "B"  SYSTEM  OK." 

♦HEARS  F/E,  "UNDING  CHECKLIST  COMPLETE." 

♦CAPTAIN  ACKNOWLEDGES. 


01:22:20  LOCALIZER  CAPTURE 


*F/0  CALLS,  "LOCALIZER  ALIVE." 

*SEES  LOCALIZER  IN  HSI  AND  ADI  IS  CENTERED. 

*SEES  SPEED  IS  AT  150  KNOTS. 

*CAPT  REDUCES  POWER  TO  SLOW  TO  VREF  +  15 
*CAPT  TRIMS  RUDDER  AND  STABILIZER 
*F/0  CALLS,  "GLIDE  SLOPE  ALIVE". 

01:22:45  OUTER  MARKER 

*HEARS  OUTER  MARKER  AUDIO  SIGNAL 
*SEES  OUTER  MARKER  LIGHT  FLASHING. 

*HEARS  F/O,  "OUTER  MARKER,  ALTIMETERS  AND 
INSTRUMENTS  CROSSCHECKED." 

*HEARS  F/O,  SACRAMENTO  TOWER,  103,  OUTER  MARKER  * 

INBOUND. 


01:23:10  LANDING  CLEARANCE 

*HEARS  SACRAMENTO  TOWER,  “  103,  SACRAMENTO 

TOWER,  WIND  340  AT  10,  CLEAREUTO  LAND  ILS  34. 
EMERGENCY  EQUIPMENT  STANDING  BY. 

*HEARS  F/O,  "103,  CLEARED  TO  LAND." 

01:23:20  CONTINUOUS  APPROACH  MONITOR 


*MONITORS  FD,  ILS  RAW  DATA,  AIRSPEED,  ALTITUDE,  RADAR 
ALTITUDE,  TIME,  VERTICAL  VELOCITY,  HSI  ILS  DISPLAY, 
HEADING,  AP  STATUS. 

01 :23:30  1000  FOOT  ALTITUDE  CALL 


*HEARS  F/O,  "1000  FEET  ABOVE  FIELD,  ALTIMETERS  AND 
INSTRUMENTS  CROSSCHECKED. 

*SEES  DESCENDING  THROUGH  1000  FT  ABOVE  FIELD. 

*CALLS,  “CROSSCHECKED."01 :24:08  500  FOOT  ALTITUDE  CALL 


*HEARS  F/O,  "500  FEET  ABOVE  FIELD,  ALTIMETERS  AND 
INSTRUMENTS  CROSSCHECKED 
*SEES  DESCENDING  THROUGH  500  FEET  ABOVE  FIELD. 
*CALLS,  "CROSSCHECKED." 


01 :24:30  VISUAL  CONTACT 

"HEARS  F/O,  ”100  FEET  TO  MINIMUMS,  STROBE  LIGHTS  IN 
SIGHT." 

*HEARS  MIDDLE  MARKER  AUDIO  SIGNAL. 

*SEES  MIDDLE  MARKER  LIGHT  FLASHING. 

*HEARS  F/O,  "MINIMUMS,  RUNWAY  IN  SIGHT." 

*SEES  MINIMUM  DESCENT  ALTITUDE  LIGHT  IS  ON. 
*LOOKS  AHEAD. 

*SEES  RUNWAY. 

*SAYS,  "CONTACT." 
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01 :25:00  TOUCHDOWN  AND  ROLLOUT 


♦SEES  TOUCHDOWN 
♦THRUST  LEVERS  IDLE. 

♦HEARS  SPEEDBRAKE  LEVER  MOVE  TO  AFT  POSITION. 
♦SENSES  AUTO  BRAKES  ARE  WORKING. 

♦SEES  GROUND  SPEED  IS  DECREASING. 

♦CONTROLS  AIRPLANE  ALONG  RUNWAY  CENTERLINE  AND 
LOWERS  NOSE  GEAR  TO  RUNWAY  SURFACE. 

♦RAISES  REVERSE  LEVERS  TO  INTERLOCK  STOP. 

♦FEELS  INTERLOCK  RELEASE. 

♦MOVES  REVERSE  LEVERS  TO  AFT  LIMIT  STOP. 

♦HEARS  F/O,  M70  KNOTS.1' 

♦MOVES  REVERSE  LEVERS  FORWARD  AND  DOWN  OUT  OF 
REVERSE  THRUST. 

♦SEES  ENGINES  AT  IDLE. 

♦PRESSES  FEET  ON  TOP  OF  BRAKE  PEDALS. 

♦CONTINUES  FINAL  TAXI  MANUALLY. 

♦HEARS  TOWER, "  103,  HOLD  ON  THE  RUNWAY 

STAY  WITH  ME. - 
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APPENDIX  C 

FULL-MISSION  SIMULATION  ATC  SCRIPTS 
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FULL-MISSION  SIMULATION  ATC  SCRIPTS 


SCRIPT  1  —  SMFtoSFO 


FREQUENCIES: 


SMF  ATIS 
SMF  TOWER 
SAC  DEPARTURE 
DISPATCH  FREQ. 


OTHER  TRAFFIC: 


N56M:  *  VISUAL  TARGET  -  LIGHT  TWIN  SOUTH  OF  SAC  VOR 
N300L:  *  VISUAL  TARGET  -  LIGHT  TWIN  NORTH  OF  SJC  VOR 


WEATHER: 

SMF  ATIS:  INFORMATION  ALPHA.  CLEAR,  VISIBILITY  15,  WIND  190/15, 

TEMPERATURE  60,  DEW  POINT  52,  ALTIMETER  29.98.  LANDING  AND 
DEPARTING  RUNWAY  16. 

SFO  ATIS:  INFORMATION  BRAVO.  CLEAR,  VISIBILITY  20,  WIND  250/5, 

TEMPERATURE  59,  DEW  POINT  45,  ALTIMETER  29.95.  LANDING  AND 
DEPARTING  RUNWAYS  28. 

MALFUNCTIONS: 


CAPTAIN  MUST  HANDLE  ALL  COMMUNICATIONS 


XX352  IS  AT  TAKEOFF  END  OF  RUNWAY  16.  THEY  HAVE  A  CLEARANCE  TO  SFO  VIA 
1  DIRECT  SAC,  RISTI  2  ARRIVAL,  CEDES,  SFO.  MAINTAIN  1 1 ,000.  DEPARTURE 

CONTROL  FREQUENCY  124.5,  SQUAWK  6512. 


XX352:  SMF  TOWER,  XX352  READY  FOR  TAKEOFF 

SMF  TOWER:  XX352,  SMF  TOWER,  CLEARED  FOR  TAKEOFF 


126.75  OAK  CENTER  132.65 

125.7  BAY  APPROACH  134.5 

124.5  SFO  TOWER  120.5 

123.55  SFO  ATIS  118.85 


WINDOW  ONE: 


OPEN  -  WHEN  THROTTLES  ADVANCE 
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CLOSE  -  FLAPS  FIVE 


SMF  TOWER:  XX352,  CONTACT  DEPARTURE  CONTROL  (1 24.5) 

(after  XX352  calls 

CLOSE  WINDOW  ON  XX352  GIVE  TIME  EFFORT  STRESS 

SAC  DEP:  XX352,  SAC  DEPARTURE,  RADAR  CONTACT 

v 


WINDOW  TWO: 


OPEN  -  FLAPS  UP 

CLOSE  -  ONE  MINUTE  LATER 


SAC  DEP:  TW789,  CONTACT  OAK  CENTER  ON  1 32.65 

TW789:  132.65,  TW789 

SAC  DEP:  XX356,  CONTACT  OAK  CENTER  ON  1 32.65 

SAC  DEP:  DL334,  CONTACT  OAK  CENTER  ON  132.65 

DL334:  OAK  CENTER  ON  1 32.65,  ROGER  334 

XX356:  CENTER  ON  132.65,  XX356 

SAC  DEP:  XX352,  LEAVING  10,000,  DO  NOT  EXCEED  300  KTS.  FOR 

SPACING 

CLOSE  WINDOW  TWO  XX352  GIVE  TIME  EFFORT  STRESS 


BETWEEN  WINDOWS  2  &  3,  APPROX  2-3  MINUTES 
(between  7,000  &  8,000  ft.) 

SAC  DEP:  XX352,  CONTACT  OAK  CENTER  ON  132.65 

OAK  CENTER:  DL334,  CLEARED  TO  SFO  VIA  THE  RISTI  2  ARRIVAL,  MAINTAIN 

11,000 


DL334: 


CLEARED  VIA  THE  RISTI  2,  MAINTAIN  1 1 ,000  ,  DL334 


OAK  CENTER: 
DL334: 

OAK  CENTER: 
N72T: 

OAK  CENTER: 
N59B: 

CESSNA  37UL 

OAK  CENTER: 
OAK  CENTER: 
OAK  CENTER: 
N72T: 

OAK  CENTER: 
N37U: 

OAK  CENTER: 


DL334  TRAFFIC,  2  O'CLOCK,  6  MILES,  VFR  9,500 
DL334,  LOOKING 

N72T,  TRAFFIC,  12  O'CLOCK,  5  MILES,  CIVIL  JET  AT  1 1 ,000 
IN  SIGHT,  XXN72T 

N59B,  CONTACT  TRAVIS  APPROACH,  127.8 
TRAVIS  APPROACH  ON  127.8,  58B 

OAKLAND  CENTER,  THIS  CESSNA  37U,  WE  RE  VFR  AT  7,500, 
REQUESTING  ADVISORIES  TO  FAT 

CESSNA  37U,  ROGER,  SQUAWK  341 2 

CESSNA  37U,  RADAR  CONTACT,  3  NORTHWEST  OF  TRACY 

N72T,  CLEAR  OF  TRAFFIC,  CONTACT  BAY  APPROACH,  120.9 

BAY  APPROACH,  120.9,  72T 

37U,  CONTACT  SCK  APPROACH,  125.1 

SCK  APPROACH,  125.1,  GOOD  DAY 

XX352,  ROGER 


WINDOW  THREE: 

OPEN  -  ONE  MINUTE  AFTER  PASSING  10,000 

CLOSE  -  TWO  MINUTES  LATER 

OAK  CENTER: 

XX352,  CLEARED  TO  SFO  VIA  THE  RISTI  2  ARRIVAL, 
MAINTAIN  11,000 

OAK  CENTER: 

DL334,  CONTACT  BAY  APPROACH  ON  134.5 

DL334: 

134.5,  DL334 

OAK  CENTER: 

XX356,  SAY  YOUR  SPEED 

XX356: 

(XXX)  KNOTS 

OAK  CENTER: 

XX352,  TRAFFIC  (XX)  O'CLOCK,  (XX)  MILES, 
(DIRECTION)  RESTRICTED  BELOW  YOU 

OAK  CENTER: 

TW789,  CONTACT  BAY  APPROACH  ON  134.5 

TW789: 


ROGER,  BAY  ON  134.5,  TW789 


OAK  CENTER:  N56M,  TRAFFIC  (XX)  O'CLOCK,  (XX)  MILES, 

(DIRECTION),  ABOVE  YOU 

N56M:  IN  SIGHT,  56M 

OAK  CENTER:  XX356,  CONTACT  BAY  APPROACH  ON  134.5 

XX356:  134.5,  XX356 

CLOSE  WINDOW  THREE  XX352  GIVE  TIME  EFFORT  STRESS 


OAK  CENTER: 
OAK  CENTER: 
PS751: 

XX787: 

OAK  CENTER: 
OAK  CENTER: 
PS751 : 

OAK  CENTER: 
OAK  CENTER: 
UA388: 

OAK  CENTER: 

UA388: 

N56M: 


BETWEEN  WINDOWS  3  &  4,  APPROX  2  -  4  MINUTES 
N56M,  CONTACT  TRAVIS  APPROACH  ON  127.8 
PS751 ,  DESCEND  AND  MAINTAIN  1 1 ,000,  SFO  ALTIMETER  29.95 
DOWN  TO  11,000,  ALTIMETER  29.95,  PS751 
OAKLAND  CENTER,  XX787  WITH  YOU  DESCENDING  TO  FL240 
XX787,  OAKLAND  CENTER,  ROGER 
PS751,  REDUCE  SPEED  TO  250 
PS751,  SLOWING  TO  250 

NAVY441,  DESCEND  AND  MAINTAIN  12,000  ,  NGZ  ALT.  29.93 
NAVY441 ,  CONTACT  BAY  APPROACH,  310.8 
OAKLAND  CENTER,  UA388  LEVEL  11,000 

UA388,  OAKLAND  CENTER,  ROGER,  CLEARED  TO  SFO,  VIA  RISTI 
2  ARRIVAL,  MAINTAIN  1 1 ,000 

TO  SFO,  VIA  THE  RISTI  2,  MAINTAIN  11,000,  UA388 

ROGER,  TRAVIS  ON  127.8 


WINDOW  FOUR: 


OPEN  -  THREE  MINUTES  AFTER  THREE  CLOSES 
CLOSE  -  TWO  MINUTES  LATER 
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BAY  APP: 

DL334,  (XX)  MILES  FROM  BRIJJ,  MAINTAIN  3,000  UNTIL 

ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  ILS  RUNWAY 

28R  APPROACH.  CONTACT  THE  TOWER  AT  BRIJJ 

BAY  APP: 

XX352,  DESCEND  AND  MAINTAIN  6,000 

DL334: 

3,000  UNTIL  ESTABLISHED,  CLEARED  FOR  APPROACH  DL334 

BAY  APP: 

TW789,  (XX)  MILES  FROM  BRIJJ,  MAINTAIN  3,000  UNTIL 
ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  ILS  RUNWAY 

28R  APPROACH. 

BAY  APP: 

XX352,  YOUR  FOLLOWING  COMPANY  TRAFFIC  10  MILES  AHEAD 

TW789: 

CLEARED  FOR  THE  ILS  TO  28R,  TW789 

V 

BAY  APP: 

CONTACT  THE  TOWER  AT  BRIJJ 

TW789: 

ROGER,  TOWER  AT  BRIJJ 

BAY  APP: 

XX356,  DESCEND  AND  MAINTAIN  4,000 

1 

XX356: 

4,000  XX356 

BAY  APP: 

XX356,  SAY  SPEED 

XX356: 

(XXX)  KNOTS 

CLOSE  WINDOW  FIVE  XX352  GIVE  TIME  EFFORT  STRESS 

BETWEEN  WINDOWS  5  &  6,  APPROX.  2-3  MINUTES 

BAY  APP: 

NAVY441,  CONTACT  NGZ  TOWER,  307.2 

PS751: 

BAY  APPROACH,  PS751  WITH  YOU  LEVELING  11,000  AT  250 

KNOTS 

t 

BAY  APP: 

PS751,  CONTINUE  DESCENT,  MAINTAIN  7,000 

PS751 : 

ROGER,  ON  DOWN  TO  7,000,  PS751 

BAY  APP: 

PS751 ,  FLY  HEADING  240,  TO  INTERCEPT  THE  28R  LOCALIZER 

PS751 : 

240  FOR  THE  INTERCEPT,  PS751 

BAY  APP: 

N4770,  SAY  AGAIN  YOUR  DESTINATION? 

N4770: 

WE'RE  GOING  TO  HAYWARD,  N4770 
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WINDOW  SIX: 


OPEN  -  LOCALIZER  ALIVE 
CLOSE  -  PASSING  LOM 


BAY  APP: 
XX356: 
BAY  APP: 
BAY  APP: 

XX356: 
AA527: 
BAY  APP: 
AA527: 

AA527: 
BAY  APP: 

AA527: 
BAY  APP: 

NW248: 
BAY  APP: 
BAY  APP: 
NW248: 
BAY  APP: 
BAY  APP: 
AA527: 


XX356,  DESCEND  AND  MAINTAIN  2,000 

ROGER,  2,000  XX356 

XX352,  DESCEND  AND  MAINTAIN  4,000 

XX356,  (XX)  MILES  FROM  BRIJJ,  MAINTAIN  2,000  UNTIL 
ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  ILS  RUNWAY 
28R  APPROACH.  CONTACT  THE  TOWER  AT  BRIJJ 

CLEARED  FOR  APPROACH,  XX356 

BAY  APPROACH  THIS  IS  AA527  WITH  YOU  AT  1 1 ,000 

AA527  BAY  APPROACH  ROGER,  DESCEND  AND  MAINTAIN  7,000 

OUT  OF  1 1 ,000  FOR  7,000  AA527  AA527,  DO  YOU  HAVE  THE 
ATIS? 

ROGER,  WE  HAVE  BRAVO 

AA527,  TURN  RIGHT  HEADING  250,  INTERCEPT  THE  28R 
LOCALIZER 

250  AND  INTERCEPT,  AA527 

XX352  (XX)  MILES  FROM  BRIJJ,  CLEARED  FOR  THE  ILS  28R 
APPROACH 

BAY  NW248  OUT  OF  7,000  FOR  1 1 ,000  REQUESTING  DIRECT  LIN 

NW248  BAY  ROGER  I'LL  CHECK 

NW248  UNABLE  DIRECT  LIN  AT  THIS  TIME 

NW248  ROGER 

AA527,  DESCEND  AND  MAINTAIN  4,000 

XX352,  SAY  SPEED 

ROGER,  DOWN  TO  4,000,  527 


CLOSE  WINDOW  SIX 


XX352  GIVE  TIME  EFFORT  STRESS 


BAY  APP:  XX352  CONTACT  TOWER  1 20.5 


SFO  TOWER: 
SFOTOWERI: 
TW789I 
SFO  TOWERI 
XX356I 


(after  XX352  calls) 

XX352  SFO  TOWER,  CLEARED  TO  LAND  28R 

TW789  TURN  LEFT  NEXT  TAXIWAY  CONTACT  GROUND  ON  .65 

ROGER 

XX356  LEFT  AT  THE  HIGH  SPEED,  GROUND  .65 
ROGER 


WINDOW  SEVEN: 


OPEN  -  PASSING  LMM 

CLOSE  -  THIRTY  SECONDS  LATER 


SFO  TOWER:  XX675,  HOLD  SHORT  OF  28R 

XX675:  HOLDING  SHORT 

CLOSE  WINDOW  SEVEN  XX352  GIVE  TIME  EFFORT  STRESS 


v 


* 


(after  XX352  has  landed) 

SFO  TOWER:  XX352,  HOLD  ON  THE  RUNWAY,  STAY  WITH  ME 


t 


J 
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SCRIPT  2  —  SFOtoSCK 


FREQUENCIES: 

SFO  ATIS 

118.85 

SCK  APPROACH 

125.1 

SFO  TOWER 

120.5 

SCK  TOWER 

120.3 

BAY  DEPARTURE 

120.9 

SCK  ATIS 

118.25 

DISPATCH  FREQ. 

123.55 

OTHER  TRAFFIC: 

N24X:  *  VISUAL  TARGET  -  LIGHT  TWIN  NORTH  OF  OAK 

N176B:  *  VISUAL  TARGET  -  LIGHT  TWIN  WEST  OF  MOD 

WEATHER: 

SFO  ATIS:  INFORMATION  BRAVO.  CLEAR,  VISIBILITY  20,  WIND 

250/5, 

TEMPERATURE  59,  DEW  POINT  45,  ALTIMETER  29.95.  LANDING  AND 
DEPARTING  28. 


SCK  ATIS:  INFORMATION  FOXTROT.  CLEAR,  VISIBILITY  20,  WIND  290/10+15, 
TEMPERATURE  62,  DEW  POINT  40,  ALTIMETER29.90.  LANDING  AND 
DEPARTING  29R 


MALFUNCTIONS: 

NONE 


XX247  IS  AT  TAKEOFF  END  OF  28L.  THEY  HAVE  A  CLEARANCE  TO  SCK  VIA  QUIET  8 
TO  REBAS,  MOD,  SIMMS,  SCK.  MAINTAIN  11,000.  DEPARTURE  FREQUENCY  120  9 
SQUAWK  3642. 


XX247:  SFO  TOWER,  THIS  IS  XX247,  READY  FOR  TAKEOFF 

SFO  TOWER:  XX247  SFO  TOWER,  CLEARED  FOR  TAKEOFF 


WINDOW  ONE: 


OPEN  -  THROTTLES  ADVANCE 
CLOSE  -  FLAPS  FIVE 
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SFO  TOWER: 


XX247,  CONTACT  BAY  DEPARTURE  (120.9) 
(after  XX247  calls) 


CLOSE  WINDOW  ONE  XX247  GIVE  TIME  EFFORT  STRESS 


BAY  DEP:  XX247,  BAY  DEPARTURE,  RADAR  CONTACT 


OPEN  -  FLAPS  UP 

WINDOW  TWO:  *> 

CLOSE  -  ONE  MINUTE  LATER 


PA454: 
BAY  DEP: 
BAY  DEP: 
EA500: 
BAY  DEP: 
EA500: 
CP422: 

BAY  DEP: 

CP422: 


BAY,  THIS  IS  PA454  OFF  OAK,  OUT  OF  1 ,000  FOR  FL230 

PA454,  THIS  IS  BAY  DEPARTURE  ROGER,  RADAR  CONTACT 

XX247,  SAY  YOUR  SPEED 

BAY,  EA500  IS  LEVEL  AT  7,000 

EA500  ROGER,  CLIMB  AND  MAINTAIN  FL230 

UP  TO  230,  EA500  THANK  YOU 

DEPARTURE  THIS  IS  CP422  OUT  OF  (XXX)  FOR  230 
REQUESTING  DIRECT  LIN 

CP422,  BAY  DEPARTURE,  RADAR  CONTACT,  UNABLE  DIRECT 
LIN 

ROGER 


CLOSE  WINDOW  TWO 


XX247  GIVE  TIME  EFFORT  STRESS 


* 


I 


J 

BAY  DEP:  XX241 ,  VERIFY  LEVEL  AT  1 1 ,000 

XX241 :  THATS  AFFIRMATIVE,  LEVEL  1 1 ,000,  XX241 

BAY  DEP:  XX247,  TRAFFIC  (XX)  O'CLOCK,  (X)  MILES,  SOUTHBOUND 

BAY  DEP:  N24X,  TRAFFIC  (XX)  O'CLOCK,  (X)  MILES,  EASTBOUND 
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N24X: 


IN  SIGHT,  THANK  YOU,  24X 


WINDOW  THREE: 


OPEN  •  ONE  MINUTE  AFTER  PASSING  10,000 
CLOSE  -  TWO  MINUTES  LATER 


* 


I 


V 


BAY  DEP:  PA454,  CONTACT  OAK  CENTER  ON  1 35.45 

PA454:  135.45,  SO  LONG 

BAY  DEP:  XX241 ,  CONTACT  SCK  APPROACH  ON  1 25.1 

XX241:  SWITCHING,  XX241 

BAY  DEP:  XX247,  DO  NOT  EXCEED  300  KTS.  FOR  SPACING 

BAY  DEP:  EA500,  CONTACT  OAK  CENTER  ON  1 28.45 

EA500:  SWITCHING,  500 

BAY  DEP:  CP422,  FLY  HEADING  040,  RECEIVING  LIN  PROCEED  DIRECT 

CP422:  040  FOR  LIN,  THANK  YOU  SIR 

BAY  DEP:  CP422,  CONTACT  OAK  CENTER  ON  1 32.65,  GOOD  DAY 

CP422:  SWITCHING 

BAY  DEP:  XX247,  CONTACT  SCK  APPROACH  ON  1 25. 1 

SCK  APP:  XX247  SCK  APPROACH,  ROGER 

CLOSE  WINDOW  THREE  XX247  GIVE  TIME  EFFORT  STRESS 


BETWEEN  WINDOWS  3  &  4,  3  MINUTES 

BARON22B:  SCK  APPROACH,  THIS  IS  BARON  22B,  JUST  OFF  LODI,  OUT  OF 

700  CLIMBING,  REQUESTING  ADVISORIES  TO  FAT 

SCK  APP:  BARON22B,  ROGER,  SQUAWK  4133 

N22B:  SQUAWKING  4133,  22B 

SCK  APP:  BARON22B,  RADAR  CONTACT  6  NORTHEAST  LODI,  SCK  ALT. 

29.90 
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BAR0N2B: 


ROGER,  29.90 

SCK  APP:  MAC1 2378,  CONTACT  TRAVIS  APPROACH,  289. 1 

SCK  APP:  AF70287,  CONTACT  CASTLE  APPROACH,  257.9 

SCK  APP:  BUFFY1 1 ,  CLIMB  AND  MAINTAIN  FL230 

SCK  APP:  BUFFY1 1 ,  CONTACT  OAKLAND  CENTER,  284.6 


OPEN  -  THREE  MINUTES  AFTER  WINDOW  THREE  CLOSES 

WINDOW  FOUR: 

CLOSE  -  TWO  MINUTES  LATER  * 


SCK  APP:  XX241 ,  DESCEND  AND  MAINTAIN  6,000 

XX241:  ROGER  OUT  OF  11,000  FOR  6,000,  XX241 

SCK  APP:  XX247,  YOUR  FOLLOWING  COMPANY  TRAFFIC  15  MILES  AHEAD 

SCK  APP:  XX241 ,  TURN  LEFT  HEADING  070 

XX241 :  LEFT  TO  070,  XX241 

SCK  APP:  XX24 1 ,  TRAFFIC  1 2  O'CLOCK,  4  MILES  SOUTHBOUND  ALTITUDE 

UNKNOWN 

XX241:  LOOKING 

SCK  APP:  PS1 278,  CONTACT  OAK  CENTER  ON  1 35.45 

PS1278:  135.45,  PS1278 

C0225:  SCK  APPROACH  C0225  WITH  YOU  AT  1 1 ,000 

SCK  APP:  C0225  SCK  APPROACH  ROGER 


\ 


I 


CLOSE  WINDOW  FOUR  XX247  GIVE  TIME  EFFORT  STRESS 

BETWEEN  WINDOW  4  &  5,  APPROX.  2-3  MINUTES 

SCK  APP:  BARON22B,  CONTACT  CASTLE  APPROACH,  1 25.2 

BARON22B:  ROGER,  CASTLE  APPROACH,  1 25.2,  BARON22B 
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4 


I 


SCKAPP: 
CITATION37L: 
SCK  APP: 
CITATION37U: 
SCK  APP: 
N2370J: 
SCKAPPI 
N2370J: 

SCK  APP: 
N233561 : 

SCK  APP: 


CITATION  37L,  CLIMB  AND  MAINTAIN  FL230 
CLIMB  AND  MAINTAIN  FL230,  CITATION  37L 
CITATION  37L,  CONTACT  OAKLAND  CENTER,  124.2 
OAKLAND  CENTER  ON  124.2,  CITATION  37U 
CHEROKEE  2370J,  CLIMB  AND  MAINTAIN  5,000 
OUT  OF  2,700  FOR  5,000  ,  CHEROKEE  70J 
CHEROKEE  70J,  CONTACT  BAY  APPROACH,  135.4 
CONTACT  BAY  ON  135.4  ,  CHEROKEE  70J 
CESSNA  23561,  CONTACT  SCK  TOWER,  120.3 
TOWER  ON  120.3  ,  CESSNA  561 

XX247  DESCEND  TO  CROSS  8  WEST  OF  MOD  AT  AND  MAINTAIN 

6,000 


WINDOW  FIVE: 


OPEN  -  THROTTLES  AT  IDLE 
CLOSE  -  TWO  MINUTES  LATER 


i 

L 


SCK  APP:  XX247,  TRAFFIC  pa)  O'CLOCK,  pa)  MILES,  SOUTHBOUND, 

BELOW  YOU 

SCK  APP:  N176B,  TRAFFIC  pa)  O'CLOCK,  pa)  MILES,  EASTBOUND,  A 

B727  ABOVE  YOU 

N176B:  INSIGHT,  176B 

SCK  APP:  N176B,  CONTACT  BAY  APPROACH  ON  134.5 

N176B:  BAY  ON  134.5,  76B,  GOOD  DAY 

SCK  APP:  XX241 ,  DESCEND  AND  MAINTAIN  3,000 

XX241:  3,000,  XX241 

SCK  APP:  XX241 ,  TRAFFIC  pa)  O'CLOCK,  pa)  MILES,  SOUTHBOUND,  VFR 

AT  2,500 

XX241:  LOOKING,  XX241 

SCK  APP:  C0225  DO  YOU  HAVE  THE  SCK  ATIS? 
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C0225:  THATS  AFFIRM  WE  HAVE  FOXTROT 

SCK  APP:  XX247,  DEPART  MOD  HEADING  320  AND  INTERCEPT  THE  29R 

LOCALIZER 

CLOSE  WINDOW  FIVE  XX247  GIVE  TIME  EFFORT  STRESS 


BETWEEN  WINDOWS  5  &  6,  APPROX.  4-5  MINUTES 

SCK  APP:  CHEROKEE  32506,  CLIMB  AND  MAINTAIN  8,000,  SAC 

ALTIMETER  29.91 

N32506:  CHEROKEE  506,  OUT  OF  6,000  FOR  8,000,  ALTIMETER  29.91 

SCK  APP:  MOONEY  231  DR,  CLIMB  AND  MAINTAIN  15,000  ,  SCK 

ALTIMETER  29.90 

N231DR:  OUT  OF  9,000  FOR  15,000  ,  ALTIMETER  29.90,  MOONEY  31  DR 

SCK  APP:  CHEROKEE  506,  CONTACT  OAKLAND  CENTER,  1 24.2 

N32506:  OAKLAND  CENTER  ON  1 24.2,  CHEROKEE  506,  SO  LONG 

PS1220:  SCK  APPROACH,  THIS  PS1220  WITH  YOU  CLIMBING  TO  7,000 

SCK  APP:  PS1 220,  SCK  APPROACH,  RADAR  CONTACT,  CLIMB  AND 

MAINTAIN  1 1 ,000,  AND  VERIFY  PRESENT  ALTITUDE 

PS 1220:  WERE  OUT  OF  1,800  ,  NOW  CLIMBING  TO  1 1,000,  PS1220 

SCK  APP:  ROGER,  SAC  ALTIMETER  IS  29.91 

PS1220:  29.91 

SCK  APP:  PS1220,  CONTACT  SAC  APPROACH,  125.6 

PS1220:  125.6,  PS 1220,  BYE 


f 


WINDOW  SIX: 


OPEN  -  LOCALIZER  ALIVE  (PASSING  MOD) 
CLOSE  -  PASSING  LOM 


V 


SCK  APP:  N905T,  TRAFFIC  (XX)  O’CLOCK,  (XX)  MILES,  EASTBOUND,  A 

CIVIL  JET  ABOVE  YOU 
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N905T: 


i 


? 


i 


SCK  APP: 
XX241: 


SCK  APP: 
SCK  APP: 
C0225: 
SCK  APP: 
XX241: 
SCK  APP: 
C0225: 
SCK  APP: 
C0225: 
XX241: 

SCK  APP: 
XX241: 
SCK  APP: 

N905T: 
SCK  APP: 
C0225: 
SCK  APP: 

N905T: 
SCK  APP: 


IN  SIGHT,  905T 

XX241, 6  MILES  FROM  JOTLY,  CLEARED  FOR  ILS  29R 
APPROACH 

CLEARED  FOR  THE  APPROACH,  XX241 
(after  XX247  passes  MOD) 

XX247,  DESCEND  AND  MAINTAIN  3,000 
C0225,  DESCEND  AND  MAINTAIN  6,000 
6,000,  C0225 

XX241 ,  CONTACT  SCK  TOWER  ON  120.3 
SWITCHING,  XX241 

C0225,  FLY  HEADING  070,  VECTORS  TO  FINAL 
LEFT  TO  070  FOR  C0225 
C0225,  DESCEND  AND  MAINTAIN  4,000 
ROGER,  DOWN  TO  4,000,  C0225 

APPROACH  THIS  IS  XX241 ,  WHAT  WAS  THAT  FREQUENCY 
AGAIN 

XX241 ,  SCK  TOWER  ON  1 20.3 
OKAY,  120.3,  THANK  YOU 

N905T,  VERIFY  YOUR  DESTINATION  AND  DO  YOU  WANT  AN  IFR 
CLEARANCE? 

AFFIRMATIVE  AND  WERE  GOING  TO  SJC,  905T 
C0225,  DESCEND  AND  MAINTAIN  3,000 
3,000,  C0225 

N905T,  CONTACT  BAY  APPROACH  ON  121 .3,  THEY  HAVE  YOUR 
REQUEST  FOR  IFR 

121.3,  905T 

XX247,  (XX)  MILES  FROM  JOTLY,  CLEARED  FOR  THE  ILS  29R 
APPROACH 
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CLOSE  WINDOW  SIX 


XX247  GIVE  TIME  EFFORT  STRESS 


SCK  APP: 

SCK  TOWER: 
SCK  TOWER: 
XX241: 

SCK  TOWER: 
N52E: 

SCK  TOWER: 
N52E: 


XX247  CONTACT  TOWER  ON  120.3 
(after  XX247  calls) 

XX247  SCK  TOWER,  CLEARED  TO  LAND  29R,  WIND  290  AT  12 
XX241,  TURN  LEFT  INTERSECTION,  GROUND  ON  POINT  9 
ROGER 

N52E,  SAY  YOUR  POSITION  NOW 

WERE  8  MILES  EAST  OF  THE  AIRPORT  AT  2,000  FOR  LANDING, 
52E 

52E  ROGER,  ENTER  LEFT  TRAFFIC  RUNWAY  29,  REPORT 
DOWNWIND 

52E,  ROGER 


1 


WINDOW  SEVEN: 


OPEN  -  PASSING  LMM 

CLOSE  -  THIRTY  SECONDS  LATER 


C0225:  SCK  TOWER,  C0225  WITH  YOU  JUST  OUTSIDE  THE  MARKER 

SCK  TOWER:  C0225  ROGER,  NUMBER  TWO 


CLOSE  WINDOW  SEVEN 


XX247  GIVE  TIME  EFFORT  STRESS 


* 


(after  XX247  is  on  runway) 

SCK  TOWER:  XX247,  HOLD  ON  THE  RUNWAY,  STAY  WITH  ME 


208 


SCRIPT  3  —  LAX  to  SFO  to  OAK  to  SMF 


FREQUENCIES: 


LAXATIS 

133.8 

OAK  CENTER 

133.7 

LAX  TOWER 

133.9 

125.45 

LAX  DEPARTURE 

125.2 

132.65 

LAX  CENTER 

135.5 

BAY  APPROACH 

135.4 

125.65 

OAK  TOWER 

127.2 

133.7 

BAY  DEPARTURE 

120.9 

125.45 

SAC  APPROACH 

125.6 

DISPATCH  FREQ. 

135.95 

SMF  TOWER 

125.7 

OAKATIS 

128.5 

SMF  ATIS 

126.75 

SFO  ATIS 

118.85 

OTHER  TRAFFIC: 


DL501 : 

PS1282: 

XX370: 

*  VISUAL  TARGET  WHEN  CAB  PASSES  13,000,  1,000  FT.  BELOW 

*  VISUAL  TARGET  WHEN  CAB  IS  AT  FL310,  2,000  FT.  ABOVE 

*  VISUAL  TARGET  WHEN  CAB  IS  NORTHWEST  OF  MQO  VOR,  2,000  FT 
ABOVE  N2578J:  *  VISUAL  TARGET  WHEN  CAB  IS  LEVEL  AT  7,000,  OFF 
OAK 

WEATHER: 

LAX  ATIS: 

INFORMATION  DELTA.  CLEAR,  VISIBILITY  15.  WIND  CALM, 
TEMPERATURE  60,  DEW  POINT  40,  ALTIMETER  29.90.  LANDING  24L, 
DEPARTING  24R. 

SFO  ATIS: 

INFORMATION  ECHO.  MEASURED  CEILING  500  OVERCAST,  VISIBILITY 
1  FOG,  WIND  280/5,  TEMPERATURE  59,  DEW  POINT  55,  ALTIMETER 
29.87.  LANDING  AND  DEPARTING  RUNWAYS  28. 

OAK  ATIS: 

INFORMATION  GOLF.  200  SCATTERED  MEASURED  400  OVERCAST, 
VISIBILITY  1  FOG,  WIND  CALM,  TEMPERATURE  55  DEW  POINT  54, 
ALTIMETER  29.86.  LANDING  AND  DEPARTING  29. 

SMF  ATIS: 

INFORMATION  HOTEL.  CLEAR,  VISIBILITY  15,  WIND  340/10 
TEMPERATURE  60,  DEW  POINT  52,  ALTIMETER  30.00.  LANDING  AND 
DEPARTING  34. 

MALFUNCTIONS: 

NO  AUTOPILOT 

15  MINUTES  INTO  FLIGHT  -  "A"  SYSTEM  WARNING  LIGHT  ACTIVATED 

PASSING  OAK  LMM  and  1,500  ft.  -  #1  ENGINE  SHUT  DOWN 

AFTER  MISSED  APPROACH  AT  OAK  and  LEAVING  6,800  ft.  -  "A"  SYSTEM  FAILURE 


XXI 03  IS  AT  TAKEOFF  END  OF  24R.  THEY  HAVE  A  CLEARANCE  TO  SFO  VIA 
THE  VENTURA  9  DEPARTURE  DIRECT  RZS  J501  BIG  SUR,  BIG  SUR  PROFILE 
DESCENT  TO  SFO.  MAINTAIN  30,  EXPECT  FL310  5  MINUTES  AFTER 
DEPARTURE.  DEPARTURE  FREQUENCY  IS  125.2,  SQUAWK  3254. 


XXI 03:  LAX  TOWER,  THIS  IS  XXI 03  READY  FOR  TAKEOFF 

LAX  TOWER.  XX103  LAX  TOWER,  CLEARED  FOR  TAKEOFF 


WINDOW  ONE: 


OPEN  •  THROTTLES  ADVANCE 
CLOSE  -  FLAPS  FIVE 


(after  passing  500  feet) 

LAX  TOWER:  XX103,  CONTACT  LAX  DEPARTURE  (125.2) 

(after  XXI 03  calls) 

LAX  DEP:  XXI 03  LAX  DEP,  RADAR  CONTACT 

CLOSE  WINDOW  ONE  XXI 03  GIVE  TIME  EFFORT  STRESS 


V 

f 


WINDOW  TWO: 


OPEN  -  FLAPS  UP 

CLOSE  -  ONE  MINUTE  LATER 


i 


LAX  DEP: 

XXI 03,  TURN  RIGHT  HEADING  280,  RECEIVING  VTU  PROCEED 
DIRECT.  CLIMB  AND  MAINTAIN  FL230 

J 

LAX  DEP: 

XXI 08,  FLY  HEADING  280,  JOIN  J501 ,  RESUME  YOUR  OWN 
NAVIGATION  CLIMB  AND  MAINTAIN  FL230,  CONTACT  LAX 

CENTER  ON  135.5 

XX 108: 

JOIN  J501,  FL230,  CHANGING,  XX108 

TW344: 

DEPARTURE,  THIS  IS  TW344  OUT  OF  700  FOR  3,000 
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LAXDEP: 


i 

4 


? 


* 

V 


TW344  LAX  DEPARTURE,  RADAR  CONTACT.  CLIMB  AND 
MAINTAIN  15,000 

TW344:  15,000,  TW344 

UA1 22:  LAX  DEPARTURE  UA1 22  WITH  YOU  OUT  OF  800  FOR  3,000 

LAX  DEP:  UA1 22  LAX  DEPARTURE,  RADAR  CONTACT,  TURN  RIGHT 

HEADING  010  VECTORS  PMD,  CLIMB  AND  MAINTAIN  15,000 

UA122:  RIGHT  TO  010  AND  UP  TO  15,000,  UA122 

CLOSE  WINDOW  2  XX103  GIVE  TIME  EFFORT  STRESS 


BETWEEN  WINDOWS  2  &  3,  APPROX  10-15  MINUTES 


LAXDEP: 


LAX  CENTER: 
LAX  CENTER: 

XXI 08: 

LAX  CENTER: 

DL501: 

LAX  CENTER: 
LAX  CENTER: 


LAX  CENTER: 
LAX  CENTER: 
MX1755: 

LAX  CENTER: 


XXI 03  CONTACT  LAX  CENTER  ON  135.5 
(after  XXI 03  calls) 

XXI 03  LAX  CENTER  ROGER 

XX108,  CLIMB  AND  MAINTAIN  FL310,  CONTACT  LAX  CENTER  ON 
125.65 

UP  TO  310  AND  125.65,  SO  LONG 

DL501,  TRAFFIC  (XX)  O'CLOCK,  (XX)  MILES,  NORTHWEST 
BOUND  ABOVE  YOU 

LOOKING,  THANK  YOU  DL501 

XXI 03,  TRAFFIC  (XX)  O'CLOCK,  pa)  MILES,  SOUTHEAST  BOUND 
BELOW  YOU 

XXI 03,  CLIMB  AND  MAINTAIN  FL310,  CONTACT  LAX  CENTER  ON 
125.65 


(after  XXI 03  calls) 

XXI 03  LAX  CENTER,  FLXXX  ROGER 

MX1755,  CLEARED  DIRECT  PMD,  REST  OF  ROUTE  UNCHANGED 
DIRECT  PMD,  MX1755 

AL260,  DESCEND  AND  MAINTAIN  15,000,  THE  BUR  ALTIMETER 
29.89,  CONTACT  LAX  CENTER  ON  135.5 
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AL260: 

LAX  CENTER: 
XX 108: 

LAX  CENTER: 
LAX  CENTER: 


15,000  AND  LAX  ON  135.5,  AL260 
XX108,  SAY  YOUR  MACH  NUMBER 
.80 

ROGER 

ASPEN  72,  VERIFY  ALTITUDE 


LAX  CENTER: 
LAX  CENTER: 
CLIPPER  220: 
LAX  CENTER: 
PS1914: 

LAX  CENTER: 

AA252: 

LAX  CENTER: 
LAX  CENTER: 

C0152: 


ASPEN  72,  ROGER,  CONTACT  OAKLAND  CENTER  338.2 

CLIPPER  220,  CONTACT  LAX  CENTER  135.5 

LAX  CENTER,  AH..  135.5,  CLIPPER  220,  GOOD  DAY 

PS1914,  CLEARED  DIRECT  BSR,  MAINTAIN  FL350 

DIRECT  BSR  NOW,  AT  350,  PS  191 4 

DACO  1 1 ,  TURN  LEFT  HEADING  090,  DIRECT  PMD  WHEN 
RECEIVING 

LAX  CENTER,  AA252  AT  FL330 
AA252,  ROGER,  330 

C0152,  RADAR  SERVICE  TERMINATED,  10  NORTHEAST  OF 
BITTY,  CLEARED  TO  ENROUTE  FREQUENCIES. 

CLEARED  TO  ENROUTE,  C0152,  SO  LONG 


1 


OPEN  >  ONE  MINUTE  AFTER  FL300 

WINDOW  THREE: 

CLOSE  -  TWO  MINUTES 

-  j 


LAX  CENTER:  XX103,  TRAFFIC  (XX)  O'CLOCK,  (XX)  MILES,  EASTBOUND  AT 

FL330 

LAX  CENTER:  PS1282,  TRAFFIC  (XX)  O'CLOCK,  (XX)  MILES,  NORTHWEST 

BOUND  AT  FL310 


PS1282:  LOOKING,  1282 

LAX  CENTER:  AA600,  DESCEND  AND  MAINTAIN  FL250,  CLEARED  DIRECT  FLW, 

DIRECT  LHS 


AA600:  DOWN  TO  250,  DIRECT  FLW  DIRECT  LHS,  THANK  YOU,  AA600 
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LAX  CENTER: 


PS1282: 

LAX  CENTER: 
XX 108: 

LAX  CENTER: 
PS1282: 

LAX  CENTER: 


PS1282  CLEAR  OF  TRAFFIC 
THANK  YOU 

XXI 08,  CONTACT  OAK  CENTER  ON  133.7 
133.7,  GOOD  NIGHT 

PS1282,  CONTACT  LAX  CENTER  ON  127.35 

127.35,  SWITCHING 

XXI 03,  SAY  YOUR  MACH  NUMBER 


CLOSE  WINDOW  THREE 


XXI 03  GIVE  TIME  EFFORT  STRESS 


(company  will  call  XXI 03  via  SELCAL  and  tell  them  to  divert  to  OAK  due  to  a  power 
failure  at  SFO)  (1  to  1  1  /2  min.  after  3  closes) 


BETWEEN  WINDOWS  3  &  4,  3  MINUTES 


LAX  CENTER: 
PS1914: 

LAX  CENTER: 

DAC011: 

UA372: 

LAX  CENTER: 

UA372: 

LAX  CENTER: 

US  AIR  118: 


PS1914,  CONTACT  OAKLAND  CENTER,  133.7 

OAKLAND  ON  133.7,  PS1914,  GOOD  NIGHT 

DACO  1 1 ,  CONTACT  LAX  CENTER,  129.1 

LAX  CENTER  ON  129.1,  DACO  1 1 

LAX  CENTER,  UA372  REQUESTING  FL370 

UA372,  ROGER,  CLIMB  AND  MAINTAIN  FL370,  REPORT 
REACHING 

OUT  OF  350  FOR  370,  WELL  REPORT  REACHING,  UA372 

US  AIR  118,  FLY  HEADING  200,  UNTIL  RECEIVING  FIM  THEN 
PROCEED  DIRECT 

LEFT  TO  200,  DIRECT  WHEN  RECEIVING,  US  AIR  118 


WINDOW  FOUR: 


OPEN  -  THREE  MINUTES  AFTER  THREE  CLOSES 
CLOSE  -  TWO  MINUTES  LATER 


LAX  CENTER:  XXI 03  CONTACT  OAK  CENTER  ON  133.7 

(after  XXI 03  calls) 

OAK  CENTER:  XXI 03  OAK  CENTER,  FLXXX 

JL557:  CENTER,  THIS  JL557  WITH  YOU  AT  FL290  REQUESTING  FL310  IF 

ITS  AVAILABLE 

OAK  CENTER:  JL557  OAK  CENTER,  CLIMB  AND  MAINTAIN  FL310 

JL557:  UP  TO  310,  THANK  YOU,  JL557 

OAK  CENTER:  UA921 ,  TRAFFIC  (XX)  O'CLOCK,  (XX)  MILES,  NORTHBOUND, 

CLIMBING  OUT  OF  290  FOR  310 

UA921 :  LOOKING  921 

OAK  CENTER:  XXI 08,  YOUR  NOW  CLEARED  TO  THE  OAK  AIRPORT  VIA  THE 

SFO  RUNWAY  28  BIG  SUR  PROFILE  DESCENT.  DEPART  MENLO 
HEADING  340  AND  INTERCEPT  THE  ILS  RUNWAY  29  FINAL 
APPROACH  COURSE.  START  YOUR  DESCENT  NOW  AND 
CONTACT  THE  OAK  CENTER  ON  125.45. 

XXI 08:  THE  BIG  SUR  PROFILE  TO  OAK,  XXI 08  CHANGING 

OAK  CENTER:  NW20,  CLEARED  DIRECT  OAL,  MAINTAIN  FL330,  REST  OF 

ROUTE  REMAINS  THE  SAME 

NW20:  DIRECT  OAL,  NW20,  WE  APPRECIATE  THAT 

OAK  CENTER:  XXI 08,  CONTACT  OAK  CENTER  ON  125.45 

XXI 08:  125.45,  SO  LONG 

(after  XX 103  calls  requesting  clearance  to  OAK) 

OAK  CENTER:  XX103  ROGER,  CLEARED  TO  THE  OAK  AIRPORT  VIA  THE  SFO 

RUNWAY  28  BSR  PROFILE  DESCENT.  DEPART  MENLO  HEADING 
340  AND  INTERCEPT  THE  ILS  RUNWAY  29  FINAL  APPROACH 
COURSE.  MAINTAIN  FL310  FOR  NOW 


CLOSE  WINDOW  FOUR  XXI 03  GIVE  TIME  EFFORT  STRESS 


OAK  CENTER:  XX103,  TRAFFIC  (XX)  O'CLOCK,  pa)  MILES,  SOUTHEAST 

BOUND,  FL330 


OAK  CENTER: 


XXI 03,  START  YOUR  DESCENT  NOW 


WINDOW  FIVE: 


OPEN  -  THROTTLES  AT  IDLE 
CLOSE  -  TWO  MINUTES  LATER 


OAK  CENTER: 

OAK  CENTER: 
PA841 : 

OAK  CENTER: 
OAK  CENTER: 

AL307: 

OAK  CENTER: 
XXI 08: 

OAK  CENTER: 
XXI 08: 

OAK  CENTER: 
AL307: 

OAK  CENTER: 


XXI 03,  CONTACT  CENTER  ON  125.45 
(after  XXI 03  calls) 

XXI 03,  ROGER 

CENTER,  PA841  WITH  YOU  AT  200 
PA841  OAK  CENTER  ROGER 

AL307,  CLIMB  AND  MAINTAIN  FL230,  EXPECT  HIGHER  IN  30 
MILES 

FL230,  AL307 

XXI 08,  REDUCE  SPEED  TO  250  KNOTS  FOR  SEQUENCING 

250  ON  THE  SPEED,  108 

XX 108,  CONTACT  BAY  APPROACH  ON  134.5 

BAY  ON  134.5,  XXI 08 

AL307,  CONTACT  OAK  CENTER  ON  134.55 
CENTER  ON  134.55,  AL307 
XXI 03,  REDUCE  TO  250  KNOTS 


CLOSE  WINDOW  FIVE 


XXI 03  GIVE  TIME  EFFORT  STRESS 


(after  XXI 03  leaves  FL190) 

OAK  CENTER:  XX103,  CONTACT  BAY  APPROACH  ON  134.5 

(after  XXI 03  calls) 

BAY  APP:  XXI 03  BAY  APPROACH  ROGER,  FLY  HEADING  330,  VECTORS  TO 

OAK,  DESCEND  AND  MAINTAIN  6,000,  DO  YOU  HAVE  THE  OAK 
ATIS? 


BAY  APP: 


XXI 08,  DESCEND  AND  MAINTAIN  4,000 

XX108:  ROGER,  DOWN  TO  4,000,  XX108 

BAY  APP:  XXI 08,  TURN  LEFT  HEADING  310  AND  INTERCEPT  THE 

LOCALIZER 

XX108:  310  FOR  THE  INTERCEPT  XX108 

BAY  APP:  XXI 08,  DESCEND  AND  MAINTAIN  3,000 

XX108:  3,000,  XX108 

BAY  APP:  PS1 492,  REDUCE  SPEED  TO  1 80  KTS 

PS 1492:  BACK  TO  180,  PS1492 

BAY  APP:  WPM34  CONTACT  BAY  APPROACH  ON  346.0 

BAY  APP:  XXI 03,  DESCEND  AND  MAINTAIN  4,000 

BAY  APP:  XXI 08,  pa)  MILES  FROM  MARCE,  MAINTAIN  2,500  UNTIL 

ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  THE  ILS  29 
APPROACH 

XXI 08:  2,500  UNTIL  ESTABLISHED,  CLEARED  FOR  APPROACH,  XXI 08 

BAY  APP:  WPM32  CONTACT  BAY  APPROACH  ON  325.2 

BAY  APP:  XXI 03,  DESCEND  AND  MAINTAIN  3,000 

BAY  APP:  XXI 03,  FLY  HEADING  pOX)  AND  INTERCEPT  THE  LOCALIZER 

OPEN  -  LOCALIZER  ALIVE  (10  NW  OF  SJC) 

WINDOW  SIX: 

CLOSE  -  PASSING  LOM 

BAY  APP:  XXI 03,  pa)  MILES  FROM  MARCE,  MAINTAIN  2,500  UNTIL 

ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  THE  ILS  29 
APPROACH 

BAY  APP:  XXI 08,  CONTACT  TOWER  ON  127.2 

XXI 08:  CHANGING 

N693X:  BAY  THIS  IS  N693X,  WEVE  DECIDED  TO  GO  TO  SJC 

N693X  ROGER,  FLY  HEADING  100,  VECTORS  TO  SJC 


BAY  APP: 


N693X: 


HEADING  100,  ROGER 
BAY  APP:  N693X  CLIMB  AND  MAINTAIN  5,000 

N693X:  UP  TO  5,000,  693 

BAY  APP:  N693X,  CONTACT  BAY  ON  1 20. 1 

N693X:  120.1,  N693X 

BAY  APP:  N57U,  TRAFFIC  1 1  O'CLOCK  4  MILES  SOUTHEAST  BOUND 

N57U:  WE  HAVE  HIM,  THANK  YOU 

BAY  APP:  N57U,  SQUAWK  1200,  RADAR  SERVICE  TERMINATED, 

FREQUENCY  CHANGE  APPROVED 

N57U:  57U  THANK  YOU,  GOOD  DAY 

(within  three  minutes  after  window  six  opens) 

BAY  APP:  XXI 03,  CONTACT  TOWER  ON  1 27.2 

(after  XXI 03  calls) 

OAK  TOWER:  XX103  OAK  TOWER,  CLEARED  TO  LAND  RUNWAY  29,  THE  RVR 

IS  NOW  2,000  BUT  IT  IS  UP  AND  DOWN,  WIND  CALM 

OAK  TOWER:  PA450,  CONTACT  GROUND  .65 

PA450:  ROGER 

(after  XXI 03  is  inside  the  outer  marker) 

CLOSE  WINDOW  SIX  XX103  GIVE  TIME  EFFORT  STRESS 


OAK  TOWER:  XXI 03,  THE  RVR  IS  LESS  THAT  1 ,000  REQUEST  INTENTIONS 


OAK  TOWER:  XXI 08,  CONTACT  BAY  DEPARTURE  ON  120.9 

XX108:  CHANGING,  XX108  * 
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CLOSE  WINDOW  SEVEN 


XXI 03  GIVE  TIME  EFFORT  STRESS 


(after  XXI 03  passes  2,000  feet  on  missed  approach) 

OPEN  -  PASSING  2,000 

WINDOW  EIGHT: 

CLOSE  -  TWO  MINUTES 

OAK  TOWER:  XXI 03,  CONTACT  BAY  DEPARTURE  ON  120.9  ^ 

(after  XXI 03  calls) 

BAY  DEP:  XXI 03  BAY  DEPARTURE,  RADAR  CONTACT,  LEAVING  3,000,  J 

TURN  RIGHT  HEADING  030,  RECEIVING  SAC  PROCEED  DIRECT, 

CLIMB  AND  MAINTAIN  7,000 

BAY  DEP:  XXI 08,  CONTACT  OAK  CENTER  ON  1 32.65 

XXI 08:  132.65,  XXI 08 

BAY  DEP:  NW473,  FLY  HEADING  120  FOR  AVE,  DIRECT  WHEN  ABLE 

NW473:  DIRECT  AVE,  THANKS 

BAY  DEP:  NW473,  CONTACT  OAK  CENTER  ON  1 23.65 

NW473:  123.65  CHANGING 


CLOSE  WINDOW  EIGHT  XX103  GIVE  TIME  EFFORT  STRESS 


OPEN  -  ONE  MINUTE  AFTER  6,800  FT. 

WINDOW  NINE: 

CLOSE  -  TWO  MINUTES  LATER 


BAY  DEP:  XXI 03,  CONTACT  OAK  CENTER  ON  1 32.65 

(after  XXI 03  calls) 


OAK  CENTER: 

XXI 03,  OAK  CENTER  ROGER 

OAK  CENTER: 

XXI 08,  CONTACT  SAC  APPROACH  ON  125.6 

XX 108: 

ROGER,  125.6,  SO  LONG 

OAK  CENTER: 

PS1254,  CLIMB  AND  MAINTAIN  FL  230 

PS 1254: 

230,  1254 

OAK  CENTER: 

PI670,  CONTACT  THE  CENTER  ON  135.45 

PI670: 

135.45,670 

i 

CLOSE  WINDOW  NINE  XXI 03  GIVE  TIME  EFFORT  STRESS 

r 

(three  to  four  minutes  before  10  opens) 

OAK  CENTER: 

XXI 03,  CONTACT  SAC  APPROACH  ON  125.6 

(after  XXI 03  calls) 

SAC  APP: 

XXI 03  SAC  APPROACH  ROGER,  DESCEND  AND  MAINTAIN  5,000 

SAC  APP: 

N498J,  TURN  LEFT  HEADING  120  FOR  TRAFFIC 

N498J: 

120  ROGER,  N498J 

SAC  APP: 

XXI 08,  (XX)  MILES  FROM  LANEE,  MAINTAIN  3,000  UNTIL 
ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  THE  ILS  34 
APPROACH 

XXI 08: 

3,000  UNTIL  ESTABLISHED,  CLEARED  FOR  APPROACH,  XX108 

L 

WINDOW  TEN: 

OPEN  -  LOCALIZER  ALIVE  (15  SE  OF  SAC) 

CLOSES  -  PASSING  LOM 

{ 

SAC  APP: 

XXI 03,  TURN  LEFT  HEADING  XXX  AND  INTERCEPT  THE  34 
LOCALIZER  DESCEND  AND  MAINTAIN  3,000 

SAC  APP: 

N498J,  TRAFFIC  NO  LONGER  A  FACTOR,  TURN  RIGHT  HEADING 
140,  PROCEED  DIRECT  MOD,  RESUME  YOUR  OWN  NAVIGATION 

N498J: 

DIRECT  MOD,  THANK  YOU,  N498J 
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SAC  APP: 

SAC  APP: 
XXI 08: 
SAC  APP: 
N498J: 
SAC  APP: 
N498J: 
SAC  APP: 


XXI 03,  (XX)  MILES  FROM  LANEE,  MAINTAIN  3,000  UNTIL 
ESTABLISHED  ON  THE  LOCALIZER,  CLEARED  FOR  THE  ILS  34 
APPROACH 

XXI 08,  CONTACT  SMF  TOWER  ON  125.7 
CHANGING 

N498J,  CLIMB  AND  MAINTAIN  FL  230 
FL  230  ROGER 

N498J,  CONTACT  OAK  CENTER  ON  132.65 
132.65,  N498J 
XXI 03,  SAY  SPEED 


CLOSE  WINDOW  TEN 


XXI 03  GIVE  TIME  EFFORT  STRESS 


SAC  APP:  XXI 03,  CONTACT  SMF  TOWER  ON  125.7 

(after  XXI 03  calls) 

SMF  TOWER:  XXI 03  SMF  TOWER  WIND  340  AT  10,  CLEARED  TO  LAND, 

EMERGENCY  EQUIPMENT  IS  STANDING  BY 


WINDOW  ELEVEN: 


OPEN  -  PASSING  LMM 

CLOSE  -  THIRTY  SECONDS  LATER 


SMF  TOWER:  UA553,  CONTACT  DEPARTURE 

UA553:  CHANGING 

(after  XXI 03  lands) 

CLOSE  WINDOW  ELEVEN  XXI 03  GIVE  TIME  EFFORT  STRESS 


SMF  TOWER:  XX103,  HOLD  ON  THE  RUNWAY,  STAY  WITH  ME 
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